-
Consumer group coordinator in Kafka using some scala script
Morning, Just a small post regarding returning a consumer group coordinator for a specific consumer group. We had the issue that consumer groups are re-balancing and we didn’t knew if it’s related to application logic or the Consumer Group Coordinator was changing and the Kafka cluster was reassign a different one each time. So, a…
-
Apache Kafka technical logs to ELK stack
Morning, Just wanted to share with you the following article: https://www.elastic.co/blog/monitoring-kafka-with-elastic-stack-1-filebeat I will try to share my experience once i have all i need to start working on it. As far as i understood from managing Kafka for a while, monitoring is not enough, this is mandatory in order to manage you clusters in a…
-
Important considerations regarding Kafka MirrorMaker
Morning folks, Just wanted to share with you two things that i learned when playing with Kafka MirrorMaker this morning. First is that if you want to put MirrorMaker or other tool that uses Kafka on debug, all you have to do is to modify the tools-log4j.properties file from the config folder of you Apache…
-
Observer functionality for puppet zookeeper module
Morning, I know it’s been some time since i last posted but i didn’t had the time to play that much. Today i want to share with you the use case in which we needed to modify the module used for the deployment of zookeeper in order to include also observer role. The link that…
-
Memory debug by Heroku guys on Apache Kafka – nice one
Hi, I know, i should write more about my experience with Apache Kafka, have patience, it’s still building, but until then please check this article: https://blog.heroku.com/fixing-kafka-memory-leak Be aware of the things that you want to include in functionalities and code that is written beside Apache Kafka functionalities, it might get you in to trouble. I…
-
Kafka limits implementation using puppet
Morning, I keep my promise and provide you with the two simple blocks that are needed to implement limits that we discussed in article http://log-it.tech/2017/10/16/ubuntu-change-ulimit-kafka-not-ignore/ For the limits module you can use: https://forge.puppet.com/puppetlabs/limits As for the actual puppet implementation, I took the decision not to restart the service immediately. This being said, it’s dead simple…
-
Ubuntu – change ulimit for kafka, do not ignore
Hi, Wanna share with you what managed to take me half a day to clarify. I just read in the following article https://docs.confluent.io/current/kafka/deployment.html#file-descriptors-and-mmap and learned that in order to optimize kafka, you will need to also change the maximum number of open files. It is nice, but our clusters are deployed on Ubuntu and the…
-
Kafka implementation using puppet at IMWorld Bucharest 2017
Hi, I recently had a presentation on how to deploy kafka using puppet and what do you need as a minimum in order to have success in production. Here is the presentation: Hope it is useful. Cheers! Update: There is also an official version from IMWorld which you can find here: And also the article…
-
Definitive guide to Kafka, confluent edition
Hi, No technical details today. Just wanted to share with you the Definitive guide to Kafka, book provided by our dear and esteem colleagues from Confluent https://www.confluent.io/wp-content/uploads/confluent-kafka-definitive-guide-complete.pdf Thank you, it should be an interesting read. Cheers!
-
Kafka problem that wasn’t a problem after all
Hi, Do not make my mistake from the last couple of weeks trying to connect to a “secured” kafka cluster using TLS. I wrote following article http://log-it.tech/2017/07/27/configure-kafka-truststore-keystore-using-puppet/ some time ago, and i know that it’s far from bullet proof but it does the job. Now let’s get to the subject, if you want to connect…