-
Delete corrupted Kafka topic version 2.0
Hi, We had in the past the situation described in this link use-case-deleting-corrupted-kafka-topic The situation repeated a little bit different this time. Taking a look on the list of topics, there were three topics marked for deletion. None of them had a Leader or Isr, so after a little bit of investigation the conclusion that…
-
Kafka service problem on upgrade to version 1.1.0
Hi, If you are using version 1.1.0 or want to upgrade to it, and the method is by puppet module provided from voxpopuli, please be aware of this issue. In the template used for the init script that it’s located under /etc/init.d/kafka and as you can also see on the latest version below: https://github.com/voxpupuli/puppet-kafka/blob/master/templates/init.erb There…
-
Consumer group coordinator in Kafka using some scala script
Morning, Just a small post regarding returning a consumer group coordinator for a specific consumer group. We had the issue that consumer groups are re-balancing and we didn’t knew if it’s related to application logic or the Consumer Group Coordinator was changing and the Kafka cluster was reassign a different one each time. So, a…
-
Important considerations regarding Kafka MirrorMaker
Morning folks, Just wanted to share with you two things that i learned when playing with Kafka MirrorMaker this morning. First is that if you want to put MirrorMaker or other tool that uses Kafka on debug, all you have to do is to modify the tools-log4j.properties file from the config folder of you Apache…
-
Observer functionality for puppet zookeeper module
Morning, I know it’s been some time since i last posted but i didn’t had the time to play that much. Today i want to share with you the use case in which we needed to modify the module used for the deployment of zookeeper in order to include also observer role. The link that…
-
Memory debug by Heroku guys on Apache Kafka – nice one
Hi, I know, i should write more about my experience with Apache Kafka, have patience, it’s still building, but until then please check this article: https://blog.heroku.com/fixing-kafka-memory-leak Be aware of the things that you want to include in functionalities and code that is written beside Apache Kafka functionalities, it might get you in to trouble. I…
-
Kafka limits implementation using puppet
Morning, I keep my promise and provide you with the two simple blocks that are needed to implement limits that we discussed in article http://log-it.tech/2017/10/16/ubuntu-change-ulimit-kafka-not-ignore/ For the limits module you can use: https://forge.puppet.com/puppetlabs/limits As for the actual puppet implementation, I took the decision not to restart the service immediately. This being said, it’s dead simple…
-
Ubuntu – change ulimit for kafka, do not ignore
Hi, Wanna share with you what managed to take me half a day to clarify. I just read in the following article https://docs.confluent.io/current/kafka/deployment.html#file-descriptors-and-mmap and learned that in order to optimize kafka, you will need to also change the maximum number of open files. It is nice, but our clusters are deployed on Ubuntu and the…
-
Kafka implementation using puppet at IMWorld Bucharest 2017
Hi, I recently had a presentation on how to deploy kafka using puppet and what do you need as a minimum in order to have success in production. Here is the presentation: Hope it is useful. Cheers! Update: There is also an official version from IMWorld which you can find here: And also the article…
-
Definitive guide to Kafka, confluent edition
Hi, No technical details today. Just wanted to share with you the Definitive guide to Kafka, book provided by our dear and esteem colleagues from Confluent https://www.confluent.io/wp-content/uploads/confluent-kafka-definitive-guide-complete.pdf Thank you, it should be an interesting read. Cheers!