Hi,
Let’s keep it short, you have a cluster composed of three Kafka brokers and you add another two. The details of doing that are pretty straight forward. If they are new, give them a unique broker.id and put the zoookeeper.connect string in order to know on which cluster to register. If they are already registered with a id, please keep in mind that you need to change it also in meta.properties from the logs.dir location.
So, we have now the situation:
Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null ls /brokers/ids [13, 14, 1003, 1002, 1001]
And we need to migrate one topic which has a replication factor of two and also two partitions:
Topic:test PartitionCount:2 ReplicationFactor:2 Configs:
Topic: test Partition: 0 Leader: 1003 Replicas: 1003,1001 Isr: 1001,1003
Topic: test Partition: 1 Leader: 1001 Replicas: 1001,1002 Isr: 1002,1001
In order to achieve this you will need to follow these steps:
Steps to be taken:
1 Create json with content:
{"topics": [{"topic": "test"}],
"version":1
}
2 Run the following command: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –topics-to-move-json-file topics-to-move.json –broker-list “13,14” –generate
Output:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[1003,1001]},{"topic":"test","partition":1,"replicas":[1001,1002]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[14,13]},{"topic":"test","partition":1,"replicas":[13,14]}]}
3 Create file expand-cluster-reassignment.json with following content:
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[14,13]},{"topic":"test","partition":1,"replicas":[13,14]}]}
4 Run following command for topic migration: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –reassignment-json-file expand-cluster-reassignment.json –execute
Output:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[1003,1001]},{"topic":"test","partition":1,"replicas":[1001,1002]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
5 Verify the status of migration with command: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –reassignment-json-file expand-cluster-reassignment.json –verify
Output: Status of partition reassignment: Reassignment of partition [test,0] completed successfully Reassignment of partition [test,1] completed successfully
Final status:
./kafka-topics.sh --zookeeper localhost:2181 --topic test --describe Topic:test PartitionCount:2 ReplicationFactor:2 Configs: Topic: test Partition: 0 Leader: 13 Replicas: 13,14 Isr: 14,13 Topic: test Partition: 1 Leader: 14 Replicas: 14,13 Isr: 13,14
This is actual a follow up of the steps described here:
https://kafka.apache.org/documentation.html#basic_ops_automigrate
Cheers!