hadoop環(huán)境如何部署
這篇文章主要講解了“hadoop環(huán)境如何部署”，文中的講解內(nèi)容簡(jiǎn)單清晰，易于學(xué)習(xí)與理解，下面請(qǐng)大家跟著小編的思路慢慢深入，一起來(lái)研究和學(xué)習(xí)“hadoop環(huán)境如何部署”吧！
專注于為中小企業(yè)提供成都網(wǎng)站建設(shè)、成都做網(wǎng)站服務(wù),電腦端+手機(jī)端+微信端的三站合一,更高效的管理,為中小企業(yè)潮陽(yáng)免費(fèi)做網(wǎng)站提供優(yōu)質(zhì)的服務(wù)。我們立足成都，凝聚了一批互聯(lián)網(wǎng)行業(yè)人才，有力地推動(dòng)了超過(guò)千家企業(yè)的穩(wěn)健成長(zhǎng)，幫助中小企業(yè)通過(guò)網(wǎng)站建設(shè)實(shí)現(xiàn)規(guī)模擴(kuò)充和轉(zhuǎn)變。
準(zhǔn)備工作
以下步驟要在所有節(jié)點(diǎn)上執(zhí)行
1.1修改hostname
vi /etc/sysconfig/network
1.2關(guān)閉SELinux
查看SELinux狀態(tài)getenforce
若SELinux沒(méi)有關(guān)閉，按照下述方式關(guān)閉
vi /etc/selinux/config
修改SELinux=disabled。重啟生效，可以等后面都設(shè)置完了重啟主機(jī)
1.3關(guān)閉防火墻
service iptables stop
chkconfig iptables off
chkconfig iptables --list
1.4網(wǎng)絡(luò)配置
vim /etc/sysconfig/network-scripts/ifcfg-eth0
1.5修改host
127.0.0.1 localhost#必須配置
# CDH Cluster
192.168.88.11 hadoop1
192.168.88.12 hadoop2
192.168.88.13 hadoop3
1.6配置hadoop1到hadoop2免密登錄
1.7所有節(jié)點(diǎn)配置NTP服務(wù)
集群中所有主機(jī)必須保持時(shí)間同步，如果時(shí)間相差較大會(huì)引起各種問(wèn)題。 具體思路如下：
master節(jié)點(diǎn)作為ntp服務(wù)器與外界對(duì)時(shí)中心同步時(shí)間，隨后對(duì)所有datanode節(jié)點(diǎn)提供時(shí)間同步服務(wù)。所有datanode節(jié)點(diǎn)以master節(jié)點(diǎn)為基礎(chǔ)同步時(shí)間。
所有節(jié)點(diǎn)安裝相關(guān)組件： yum install ntp 。
完成后，配置開機(jī)啟動(dòng)： chkconfig ntpd on ,
檢查是否設(shè)置成功： chkconfig --list ntpd 其中2-5為on狀態(tài)就代表成功。
主節(jié)點(diǎn)配置
在配置之前，先使用ntpdate手動(dòng)同步一下時(shí)間，免得本機(jī)與對(duì)時(shí)中心時(shí)間差距太大，使得ntpd不能正常同步。這里選用65.55.56.206作為對(duì)時(shí)中心, ntpdate -u 202.112.10.36
vi /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
# Hosts on local network are less restricted.
# 允許內(nèi)網(wǎng)其他機(jī)器同步時(shí)間
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# 中國(guó)這邊最活躍的時(shí)間服務(wù)器 : http://www.pool.ntp.org/zone/cn
server 210.72.145.44 perfer # 中國(guó)國(guó)家受時(shí)中心
server 202.112.10.36 # 1.cn.pool.ntp.org
server 59.124.196.83 # 0.asia.pool.ntp.org
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# allow update time by the upper server
# 允許上層時(shí)間服務(wù)器主動(dòng)修改本機(jī)時(shí)間
restrict 210.72.145.44 nomodify notrap noquery
restrict 202.112.10.36 nomodify notrap noquery
restrict 59.124.196.83 nomodify notrap noquery
# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
# 外部時(shí)間服務(wù)器不可用時(shí)，以本地時(shí)間作為時(shí)間服務(wù)
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
service ntpd start
ntpstat
這個(gè)一般需要5-10分鐘后才能成功連接和同步
[root@hadoop1 ~]# netstat -tlunp | grep ntp
udp 0 0 192.168.88.11:123 0.0.0.0:* 17339/ntpd ############################
udp 0 0 127.0.0.1:123 0.0.0.0:* 17339/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 17339/ntpd
udp 0 0 fe80::20c:29ff:fe7c:123 :::* 17339/ntpd
udp 0 0 ::1:123 :::* 17339/ntpd
udp 0 0 :::123 :::* 17339/ntpd
[root@hadoop1 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
202.118.1.130 .INIT. 16 u - 64 0 0.000 0.000 0.000#################################
# ntpstat
unsynchronised
time server re-starting
polling server every 64 s
連接并同步后:
synchronised to NTP server (202.112.10.36) at stratum 3
time correct to within 275 ms
polling server every 256 s
# yum install ntp
# chkconfig ntp on
# vim /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
# 配置時(shí)間服務(wù)器為本地的時(shí)間服務(wù)器
server 192.168.1.135
restrict 192.168.1.135 nomodify notrap noquery
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
[root@hadoop2 soft]# ntpdate -u hadoop1
2.cloudra安裝 所有節(jié)點(diǎn)
2.1下載cloudera-manager.repo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo
1將cloudera-manager.repo文件拷貝到所有節(jié)點(diǎn)的/etc/yum.repos.d/文件夾下
mv cloudera-manager.repo /etc/yum.repos.d/
vi /etc/yum.conf
timeout=50000
yum list|grep cloudera
如果列出的不是你安裝的版本，執(zhí)行下面命令重試
yum clean all
yum list | grep cloudera
2.2下載CDH將之前下載的Parcel那3個(gè)文件拷貝到/opt/cloudera/parcel-repo目錄下（如果沒(méi)有該目錄，請(qǐng)自行創(chuàng)建）
wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel
wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1###.sha1文件后綴更改為.sha，同時(shí)把內(nèi)容只保留hash碼部分
wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/manifest.json
2.4在master[hadoop1] 節(jié)點(diǎn)安裝daemons、server、agent（先裝daemons）
wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el5.x86_64.rpm
wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-server-5.2.1-1.cm521.p0.109.el5.x86_64.rpm
wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-agent-5.2.1-1.cm521.p0.109.el5.x86_64.rpm
yum --nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm
yum --nogpgcheck localinstall cloudera-manager-server-5.2.1-1.cm521.p0.109.el6.x86_64.rpm
yum --nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm（注：agent安裝需要聯(lián)網(wǎng)）
2.5在slave-1[hadoop2]、slave-2[hadoop3]節(jié)點(diǎn)安裝daemons、agent（先裝daemons）
wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el5.x86_64.rpm
wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-agent-5.2.1-1.cm521.p0.109.el5.x86_64.rpm
yum --nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm
yum --nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm（注：agent安裝需要聯(lián)網(wǎng)）
2.6在master、slave-1、slave-2 節(jié)點(diǎn)安裝JDK、oraclejdk
rpm -ivh jdk-6u31-linux-amd64.rpm
3.在master節(jié)點(diǎn)安裝MySQL 數(shù)據(jù)庫(kù)，并配置cdh需要的數(shù)據(jù)庫(kù)選項(xiàng)
yum install mysql-server mysql mysql-devel
chkconfig mysqld on
service mysqld start
mysql –u root
use mysql
update user set password=password('1234') where user='root'
update user set password=password('1234') where host='localhost'
update user set password=password('1234') where host='hadoop1'
service mysqld restart
mysql -u root -p1234
create database cloudera
4.在master節(jié)點(diǎn)配置cloudera manager 數(shù)據(jù)庫(kù)并啟動(dòng)cm的server及agent程序
1.拷貝mysql-connector-java-5.1.7-bin.jar 到 /usr/share/java 下并重命名mysql-connector-java.jar
2.運(yùn)行 /usr/share/cmf/schema/scm_prepare_database.sh -h hadoop1 mysql cloudera root 1234
3.啟動(dòng)cm server ：service cloudera-scm-server start
4.添加cm server服務(wù) ：chkconfig cloudera-scm-server on
5.啟動(dòng)cm agent ：chkconfig cloudera-scm-agent on
6.添加cm agent服務(wù) ：service cloudera-scm-server start
5、修改所有節(jié)點(diǎn)的agent 配置文件
/etc/cloudera-scm-agent/config.ini 將配置文件中的host 改成 cdh-master
6、在slave節(jié)點(diǎn)配置cloudera manager agent程序
1.啟動(dòng)cm agent ：chkconfig cloudera-scm-agent on
2.添加cm agent服務(wù) ：service cloudera-scm-agent start
7、測(cè)試agent和server是否通信成功
service cloudera-scm-server status
service cloudera-scm-agent status
netstat –anp | grep 7182
# server 端開啟的是7182端口，用于和agent進(jìn)行通訊
啟動(dòng)失敗時(shí)可以查看日志
server 日志 /var/log/cloudera-scm-server
agent 日志 /var/log/cloudera-scm-agent
8設(shè)置parcel[master]
mv CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel /opt/cloudera/parcel-repo
[root@hadoop1 parcel-repo]# tail -5 manifest.json
"replaces": "IMPALA, SOLR, SPARK",
"hash": "7dcb31e557a7da951bfb6337e02b0b884aa3d2a2\n"
}
]
[root@hadoop1 parcel-repo]# tail -1 CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1
7dcb31e557a7da951bfb6337e02b0b884aa3d2a2
[root@hadoop1 parcel-repo]# mv CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1 CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha
9.[root@hadoop1 soft]# rpm -ivh oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm 所有節(jié)點(diǎn)
CDH集群安裝
CM安裝成功后瀏覽器輸入http://ip:7180,ip是CM安裝的主機(jī)ip或者主機(jī)名。顯示如下界面，用戶名和密碼都輸入admin，進(jìn)入web管理界面。
免費(fèi)版-〉繼續(xù)->查找并選擇需要安裝 CDH 的機(jī)器，點(diǎn)擊“繼續(xù)” 192.168.88.[11-13]->
二、卸載步驟
記錄卸載過(guò)程和問(wèn)題。現(xiàn)有環(huán)境Cloudera Manager + (1 + 2 )的CDH環(huán)境。
1、先在Manage管理端移除所有服務(wù)。
2、刪除Manager Server
在Manager節(jié)點(diǎn)運(yùn)行
/usr/share/cmf/uninstall-cloudera-manager.sh如果沒(méi)有該腳本，則可以手動(dòng)刪除，先停止服務(wù)：
service cloudera-scm-server stop
service cloudera-scm-server-db stop然后刪除：
yum remove cloudera-manager-serversudo
yum remove cloudera-manager-server-db3 、刪除所有CDH節(jié)點(diǎn)上的CDH服務(wù)，先停止服務(wù)：
service cloudera-scm-agent hard_stop卸載安裝的軟件：
yum remove 'cloudera-manager-*' hadoop hue-common 'bigtop-*'4、刪除殘余數(shù)據(jù)：
rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera*
5、kill掉所有Manager和Hadoop進(jìn)程（選作，如果你正確停止Cloud Manager和所有服務(wù)則無(wú)須此步）
$ for u in hdfs mapred cloudera-scm hbase hue zookeeper oozie hive impala flume; do sudo kill $(ps -u $u -o pid=); done6、刪除Manager的lock文件
在Manager節(jié)點(diǎn)運(yùn)行：
rm /tmp/.scm_prepare_node.lock至此，刪除完成。
/var/log/cloudera-manager-installer/3.install-cloudera-manager-server.log
http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/
獲取鎖 卸載裝
Couldn't resolve host 'archive.cloudera.com'" DNS8.8.8.8
注意主機(jī)名要與host一致，若不一致就刪掉，重新搜索
正在搜索要重新卸載，再安裝
[root@h02 soft]# service cloudera-scm-agent status
cloudera-scm-agent dead but pid file exists
[root@client ~]# cd /var/run
[root@client ~]# rm –f cloudera-scm-agent.pid
在日志中發(fā)現(xiàn)這樣一條錯(cuò)誤信息：
ERROR ENGINE Error in HTTP server: shutting down Traceback (most recent call last)
IOError: [Errno 2] No such file or directory: '/var/lib/cloudera-scm-agent/uuid'
[root@h02 cloudera-scm-agent]# mkdir /var/lib/cloudera-scm-agent/
[root@h02 cloudera-scm-agent]# chmod 777 /var/lib/cloudera-scm-agent/
感謝各位的閱讀，以上就是“hadoop環(huán)境如何部署”的內(nèi)容了，經(jīng)過(guò)本文的學(xué)習(xí)后，相信大家對(duì)hadoop環(huán)境如何部署這一問(wèn)題有了更深刻的體會(huì)，具體使用情況還需要大家實(shí)踐驗(yàn)證。這里是創(chuàng)新互聯(lián)，小編將為大家推送更多相關(guān)知識(shí)點(diǎn)的文章，歡迎關(guān)注！
新聞標(biāo)題：hadoop環(huán)境如何部署
轉(zhuǎn)載來(lái)源：http://bm7419.com/article34/jceepe.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)站營(yíng)銷、、微信公眾號(hào)、定制開發(fā)、App設(shè)計(jì)、網(wǎng)站維護(hù)
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)
- 怎么做好網(wǎng)站導(dǎo)航SEO優(yōu)化？ 2015-09-20
- 站長(zhǎng)如何劃分清晰的網(wǎng)站導(dǎo)航欄目 2022-09-22
- 網(wǎng)站數(shù)據(jù)分析篇之網(wǎng)站導(dǎo)航分析 2021-09-13
- 良好的網(wǎng)站導(dǎo)航如何網(wǎng)站建設(shè)可用性 2016-08-23
- 從seo優(yōu)化的角度怎么設(shè)計(jì)網(wǎng)站導(dǎo)航欄 2016-09-08
- 設(shè)計(jì)有良好用戶體驗(yàn)的網(wǎng)站導(dǎo)航 2016-10-09
- 設(shè)計(jì)一個(gè)有創(chuàng)意的網(wǎng)站導(dǎo)航方法 2021-04-22
- 愛(ài)不釋手的側(cè)邊欄！網(wǎng)站導(dǎo)航應(yīng)該去一邊玩兒！ 2022-06-08
- 設(shè)計(jì)好網(wǎng)站導(dǎo)航菜單,留給用戶的印象-佛山網(wǎng)站設(shè)計(jì) 2022-11-10
- 做好網(wǎng)站導(dǎo)航SEO優(yōu)化的重要作用 2023-04-08
- 網(wǎng)站導(dǎo)航菜單教程網(wǎng)站頂部導(dǎo)航欄設(shè)計(jì) 2021-06-10
- 建網(wǎng)站良好的網(wǎng)站導(dǎo)航把握以下幾點(diǎn)原則 2015-11-08