kafka集群安裝及管理（二）

一、broker的遷移

譙城ssl適用于網(wǎng)站、小程序/APP、API接口等需要進(jìn)行數(shù)據(jù)傳輸應(yīng)用場景，ssl證書未來市場廣闊！成為創(chuàng)新互聯(lián)公司的ssl證書銷售渠道，可以享受市場價格4-6折優(yōu)惠！如果有意向歡迎電話聯(lián)系或者加微信：028-86922220（備注：SSL證書合作）期待與您的合作！

1.查看zookeeper和kafka啟動情況

[root@slave1 ~]# pssh -h hostlist -i 'jps'
[1] 22:08:11 [SUCCESS] 20.0.5.12
3492 QuorumPeerMain
8140 Jps
6414 Kafka
[2] 22:08:11 [SUCCESS] 20.0.5.13
3490 QuorumPeerMain
4972 Kafka
22972 Jps
[3] 22:08:11 [SUCCESS] 20.0.5.11
7369 QuorumPeerMain
23754 Jps
11534 Kafka
[4] 22:08:11 [SUCCESS] 20.0.5.14
21263 Jps
[5] 22:08:11 [SUCCESS] 20.0.5.15
20818 Jps

2.創(chuàng)建一個topic

[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 20.0.5.11:2181 --replication-factor 3 --partitions 3 --topic topic1
Created topic "topic1".
[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 20.0.5.11:2181 --topic topic1
Topic:topic1	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: topic1	Partition: 0	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: topic1	Partition: 1	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: topic1	Partition: 2	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1

3.在要遷移的節(jié)點上啟動kafka進(jìn)程

[root@slave1 ~]# pssh -h hostlist '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /root/kafka.log 2>1&'
[1] 22:17:21 [SUCCESS] 20.0.5.12
[2] 22:17:21 [SUCCESS] 20.0.5.11
[3] 22:17:21 [SUCCESS] 20.0.5.13
[4] 22:17:21 [SUCCESS] 20.0.5.14
[5] 22:17:21 [SUCCESS] 20.0.5.15
[root@slave1 ~]# pssh -h hostlist -i 'jps'
[1] 22:17:29 [SUCCESS] 20.0.5.12
3492 QuorumPeerMain
8461 Jps
6414 Kafka
[2] 22:17:29 [SUCCESS] 20.0.5.14
21298 Kafka
21609 Jps
[3] 22:17:29 [SUCCESS] 20.0.5.11
7369 QuorumPeerMain
24844 Jps
11534 Kafka
[4] 22:17:29 [SUCCESS] 20.0.5.13
3490 QuorumPeerMain
4972 Kafka
23292 Jps
[5] 22:17:29 [SUCCESS] 20.0.5.15
20854 Kafka
21165 Jps

4.創(chuàng)建需要遷移的topic的json文件

[root@slave1 ~]# cat topic_move.json
{"topics": [{"topic": "topic1"}],
"version":1
}

5.生成遷移規(guī)則

[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --topics-to-move-json-file topic_move.json --broker-list "3,4,5" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[5,3,4],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[4,5,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,4,5],"log_dirs":["any","any","any"]}]}

6.將生成的數(shù)據(jù)寫入新的json文件

[root@slave1 ~]# cat re_node.json
{"version":1,"partitions":[
{"topic":"topic1","partition":1,"replicas":[5,3,4],"log_dirs":["any","any","any"]},
{"topic":"topic1","partition":0,"replicas":[4,5,3],"log_dirs":["any","any","any"]},
{"topic":"topic1","partition":2,"replicas":[3,4,5],"log_dirs":["any","any","any"]}]
}

7.開始遷移

[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --reassignment-json-file re_node.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"topic1","partition":1,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":0,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"topic1","partition":2,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@slave1 ~]# /opt/kafka/bin/kafka-reassign-partitions.sh --zookeeper 20.0.5.11:2181 --reassignment-json-file re_node.json --verify
Status of partition reassignment:
Reassignment of partition topic1-1 completed successfully
Reassignment of partition topic1-0 completed successfully
Reassignment of partition topic1-2 completed successfully

8.查看新的分區(qū)

[root@slave1 ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 20.0.5.11:2181 --topic topic1
Topic:topic1	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: topic1	Partition: 0	Leader: 4	Replicas: 4,5,3	Isr: 5,3,4
	Topic: topic1	Partition: 1	Leader: 5	Replicas: 5,3,4	Isr: 5,3,4
	Topic: topic1	Partition: 2	Leader: 3	Replicas: 3,4,5	Isr: 5,3,4

#遷移完后修改好producer和consumer指向新的連接后再停止舊的服務(wù)

本文標(biāo)題：kafka集群安裝及管理（二）
轉(zhuǎn)載注明：http://bm7419.com/article34/gocepe.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供動態(tài)網(wǎng)站、定制開發(fā)、企業(yè)網(wǎng)站制作、用戶體驗、網(wǎng)站導(dǎo)航、響應(yīng)式網(wǎng)站

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)

成都app開發(fā)公司