Sqoop簡(jiǎn)介及安裝部署-創(chuàng)新互聯(lián)

簡(jiǎn)介：

成都創(chuàng)新互聯(lián)公司長(zhǎng)期為近千家客戶提供的網(wǎng)站建設(shè)服務(wù)，團(tuán)隊(duì)從業(yè)經(jīng)驗(yàn)10年，關(guān)注不同地域、不同群體，并針對(duì)不同對(duì)象提供差異化的產(chǎn)品和服務(wù)；打造開(kāi)放共贏平臺(tái)，與合作伙伴共同營(yíng)造健康的互聯(lián)網(wǎng)生態(tài)環(huán)境。為可克達(dá)拉企業(yè)提供專業(yè)的網(wǎng)站建設(shè)、做網(wǎng)站，可克達(dá)拉網(wǎng)站改版等技術(shù)服務(wù)。擁有十年豐富建站經(jīng)驗(yàn)和眾多成功案例,為您定制開(kāi)發(fā)。

Apache Sqoop是專為Apache Hadoop和結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)如關(guān)系數(shù)據(jù)庫(kù)之間的數(shù)據(jù)轉(zhuǎn)換工具的有效工具。你可以使用Sqoop從外部結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)的數(shù)據(jù)導(dǎo)入到Hadoop分布式文件系統(tǒng)或相關(guān)系統(tǒng)如Hive和HBase。相反，Sqoop可以用來(lái)從Hadoop的數(shù)據(jù)提取和導(dǎo)出到外部結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)如關(guān)系數(shù)據(jù)庫(kù)和企業(yè)數(shù)據(jù)倉(cāng)庫(kù)。
Sqoop專為大數(shù)據(jù)批量傳輸設(shè)計(jì)，能夠分割數(shù)據(jù)集并創(chuàng)建Hadoop任務(wù)來(lái)處理每個(gè)區(qū)塊。

下面介紹下安裝部署的步驟：

1.下載安裝包并解壓

下載地址作者使用的是sqoop-1.4.6-cdh6.7.0.tar.gz版本

# wget下載安裝包 也可下載到本地 然后上傳至Linux
[hadoop@hadoop000 software]$ pwd
/home/hadoop/software
[hadoop@hadoop000 software]$ wget http://archive.cloudera.com/cdh6/cdh/5/sqoop-1.4.6-cdh6.7.0.tar.gz
# 解壓
[hadoop@hadoop000 software]$ tar -xzvf sqoop-1.4.6-cdh6.7.0.tar.gz -C /home/hadoop/app/
# 查看解壓目錄
[hadoop@hadoop000 sqoop-1.4.6-cdh6.7.0]$ ls -lh
total 1.9M
drwxr-xr-x 2 hadoop hadoop  4.0K Jul 3 16:00 bin --執(zhí)行腳本目錄
-rw-rw-r-- 1 hadoop hadoop   60K Mar 24  2016 build.xml
-rw-rw-r-- 1 hadoop hadoop  1.1K Mar 24  2016 cdh.build.properties
-rw-rw-r-- 1 hadoop hadoop   35K Mar 24  2016 CHANGELOG.txt
drwxr-xr-x 4 hadoop hadoop  4.0K Jul 3 16:00 cloudera
-rw-rw-r-- 1 hadoop hadoop  6.8K Mar 24  2016 cloudera-pom.xml
-rw-rw-r-- 1 hadoop hadoop  9.7K Mar 24  2016 COMPILING.txt
drwxr-xr-x 2 hadoop hadoop  4.0K Jul 3 16:00 conf  --配置文件目錄
drwxr-xr-x 5 hadoop hadoop  4.0K Jul 3 16:00 docs  --文檔
drwxr-xr-x 2 hadoop hadoop  4.0K Jul 3 16:00 ivy
-rw-rw-r-- 1 hadoop hadoop   17K Mar 24  2016 ivy.xml
drwxr-xr-x 2 hadoop hadoop  4.0K Jul 3 16:00 lib  --lib依賴包
-rw-rw-r-- 1 hadoop hadoop   15K Mar 24  2016 LICENSE.txt
-rw-rw-r-- 1 hadoop hadoop   505 Mar 24  2016 NOTICE.txt
-rw-rw-r-- 1 hadoop hadoop   19K Mar 24  2016 pom-old.xml
-rw-rw-r-- 1 hadoop hadoop  1.1K Mar 24  2016 README.txt
-rw-rw-r-- 1 hadoop hadoop 1012K Mar 24  2016 sqoop-1.4.6-cdh6.7.0.jar  --完整jar包
-rw-rw-r-- 1 hadoop hadoop  6.5K Mar 24  2016 sqoop-patch-review.py
-rw-rw-r-- 1 hadoop hadoop  641K Mar 24  2016 sqoop-test-1.4.6-cdh6.7.0.jar
drwxr-xr-x 7 hadoop hadoop  4.0K Mar 24  2016 src  --源碼
drwxr-xr-x 4 hadoop hadoop  4.0K Jul 3 16:00 testdata

2.配置環(huán)境變量

# 添加sqoop環(huán)境變量 可加入全局 也可只配置個(gè)人環(huán)境變量
[hadoop@hadoop000 ~]$ sudo vi/etc/profile
export SQOOP_HOME=/home/hadoop/app/sqoop-1.4.6-cdh6.7.0
export PATH=$SQOOP_HOME/bin:$PATH
[hadoop@hadoop000 ~]$ source /etc/profile

3.修改sqoop配置文件

[hadoop@hadoop000 conf]$ pwd
/home/hadoop/app/sqoop-1.4.6-cdh6.7.0/conf
[hadoop@hadoop000 conf]$ cp sqoop-env-template.sh sqoop-env.sh
# 添加hadoop及hive目錄
[hadoop@hadoop000 conf]$ vi sqoop-env.sh
#Set path to where bin/hadoop is available 
export HADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0

#set the path to where bin/hbase is available
#export HBASE_HOME=

#Set the path to where bin/hive is available                                                                                    
export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh6.7.0

#Set the path for where zookeper config dir is                                                                                  
#export ZOOCFGDIR=

4.拷貝jdbc驅(qū)動(dòng)包到sqoop/lib目錄下

# 將Hive lib目錄下的mysql驅(qū)動(dòng)包拷貝過(guò)來(lái)
[hadoop@hadoop000 lib]$ pwd
/home/hadoop/app/sqoop-1.4.6-cdh6.7.0/lib
[hadoop@hadoop000 lib]$ cp /home/hadoop/app/hive-1.1.0-cdh6.7.0/lib/mysql-connector-java-5.1.46.jar .

5.sqoop簡(jiǎn)單測(cè)試使用

# 查看命令幫助
[hadoop@hadoop000 ~]$ sqoop help
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/03 16:23:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh6.7.0
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job               work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.
# 查看sqoop版本
[hadoop@hadoop000 ~]$ sqoop version
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/03 16:23:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh6.7.0
Sqoop 1.4.6-cdh6.7.0
git commit id 
Compiled by jenkins on Wed Mar 23 11:30:51 PDT 2016

# 這里的警告是因?yàn)槲覜](méi)有配置hbase，zookeeper，HCatalog

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。

分享名稱：Sqoop簡(jiǎn)介及安裝部署-創(chuàng)新互聯(lián)
本文來(lái)源：http://bm7419.com/article10/cecigo.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供定制開(kāi)發(fā)、虛擬主機(jī)、網(wǎng)站內(nèi)鏈、外貿(mào)網(wǎng)站建設(shè)、動(dòng)態(tài)網(wǎng)站、營(yíng)銷型網(wǎng)站建設(shè)

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容