Important considerations regarding Kafka MirrorMaker

Morning folks,

Just wanted to share with you two things that i learned when playing with Kafka MirrorMaker this morning.

First is that if you want to put MirrorMaker or other tool that uses Kafka on debug, all you have to do is to modify the tools-log4j.properties file from the config folder of you Apache Kafka installation. It normally looks like this

log4j.rootLogger=WARN, stderr

log4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stderr.Target=System.err

but you can easily set it to

log4j.rootLogger=DEBUG, stderr

log4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stderr.Target=System.err

The second one is related to property that you should put on your consumer.config file in order to grab the data from the topic start and not from the latest offset.

In order to achieve this you will need to add the following option:

auto.offset.reset=earliest

The possible values for this setting are latest, earliest, none.

Cheers