Automigrate data contained in a topic to different Kafka brokers

Written by

Sorin Tudor

in

Hi,

Let’s keep it short, you have a cluster composed of three Kafka brokers and you add another two. The details of doing that are pretty straight forward. If they are new, give them a unique broker.id and put the zoookeeper.connect string in order to know on which cluster to register. If they are already registered with a id, please keep in mind that you need to change it also in meta.properties from the logs.dir location.

So, we have now the situation:

Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ls /brokers/ids
[13, 14, 1003, 1002, 1001]

And we need to migrate one topic which has a replication factor of two and also two partitions:

Topic:test    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: test    Partition: 0    Leader: 1003    Replicas: 1003,1001    Isr: 1001,1003
    Topic: test    Partition: 1    Leader: 1001    Replicas: 1001,1002    Isr: 1002,1001

In order to achieve this you will need to follow these steps:

Steps to be taken:
1 Create json with content:

{"topics": [{"topic": "test"}],
  "version":1
  }

2 Run the following command: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –topics-to-move-json-file topics-to-move.json –broker-list “13,14” –generate
Output:

Current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[1003,1001]},{"topic":"test","partition":1,"replicas":[1001,1002]}]}
Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[14,13]},{"topic":"test","partition":1,"replicas":[13,14]}]}

3 Create file expand-cluster-reassignment.json with following content:

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[14,13]},{"topic":"test","partition":1,"replicas":[13,14]}]}

4 Run following command for topic migration: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –reassignment-json-file expand-cluster-reassignment.json –execute
Output:

Current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[1003,1001]},{"topic":"test","partition":1,"replicas":[1001,1002]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

5 Verify the status of migration with command: ./kafka-reassign-partitions.sh –zookeeper localhost:2181 –reassignment-json-file expand-cluster-reassignment.json –verify

Output:
Status of partition reassignment: 
Reassignment of partition [test,0] completed successfully
Reassignment of partition [test,1] completed successfully

Final status:

./kafka-topics.sh --zookeeper localhost:2181 --topic test --describe
Topic:test	PartitionCount:2	ReplicationFactor:2	Configs:
	Topic: test	Partition: 0	Leader: 13	Replicas: 13,14	Isr: 14,13
	Topic: test	Partition: 1	Leader: 14	Replicas: 14,13	Isr: 13,14

This is actual a follow up of the steps described here:

https://kafka.apache.org/documentation.html#basic_ops_automigrate

Cheers!

More posts