-
SASL config issue on latest Kafka versions
Hello, Today I want to share with you a problem that we needed to fix when we decided to activate SASL. Normally, the steps are pretty straight forward and you can use Confluent doku or the general Apache Kafka. The main catch is that if you have a certain property in your config file, the…
-
Automatic increase of Kafka LVM on GCP
I wrote an article for my company that was published on Medium regarding the topic in the subject. Please see the link https://medium.com/metrosystemsro/new-ground-automatic-increase-of-kafka-lvm-on-gcp-311633b0816c Thanks
-
Kafka_consumer.yaml (python style) and more
Hi, As a followup to the article i posted earlier ( https://log-it.tech/2019/03/15/get-the-info-you-need-from-consumer-group-python-style/ ) , you can use that info to put in into kafka_consumer.yaml for Datadog integration. It’s not elegant by any means, but it works. As an advise, please don’t over complicate thinks more than they need. In the last example i figured i…
-
Get the info you need from consumer group (python style)
Hi, For some of you might be of help. If it’s rubbish, i truly apologize, please leave a comment with improvements 🙂 I will not explain it. It should be self explanatory. Cheers
-
Kafka consumer group info retrieval using Python
Hi, I’ve been playing with kafka-python module to grab the info i need in order to reconfigure Datadog integration. Unfortunately, there is a catch also on this method. And i will show you below. Here is a little bit of not so elegant code. As you will see, print(meta[3]) will return a very ugly binary…
-
Fact for kafka_consumer hash…or kind of
Hi, There is a late requirement that we activate the kafka_consumer functionality of Datadog. Unfortunately this is a challenge if you don’t have a fixed number of consumer groups and topics (on one client we had a couple of hundreds consumer groups) Here is how it should look in the example file So, if you…
-
Small go code example for zookeeper resource editing
Hi, We have the task of “service restart coordination” for our Apache Kafka cluster. It’s still a work in progress but if you want to use the zookeeper for some status verification and update, something like this will work as an example. package main import ( “fmt” “io” “launchpad.net/gozk” “os” “strings” “sync” “time” ) const…
-
Don’t delete the Kafka GC logs when they are used
Hi, I made a mistake some time ago, and it’s there to hunt me. Deleting the normal gc logs including the one it’s already used doesn’t solve anything, it just created a more difficult situation. Here is my example: /dev/sda1 50G 42G 5.2G 90% / /opt/kafka/logs# ll total 34M drwxrwxr-x 2 kafka kafka 4.0K Oct…
-
Final version of SSL gen script for kafka
Hi, I wrote a lot about this topic but it seems that i came to the procedure specified by Confluent. Here is the right way to do it, at least for now: #!/bin/bash HOST= PASSWORD= KEYSTOREPASS= VALIDITY=365 keytool -keystore kafka.server.keystore.jks -alias ${HOST} -validity $VALIDITY -genkey -dname “CN=${HOST}, OU=MyTeam, O=MyCompany, L=Bucharest S=Romania C=RO” -storepass $KEYSTOREPASS -keypass…
-
Wrong again, there is no return code 0 on self signed certs
Morning, It looks like i was wrong again with the SSL generation script. Here is the second article Code 0 is not good after all and it signals that Kafka broker is closing the connection really fast. So: There is no 0 on self signed certs Please make sure that you have a certificate in…