-
Kafka problem that wasn’t a problem after all
Hi, Do not make my mistake from the last couple of weeks trying to connect to a “secured” kafka cluster using TLS. I wrote following article http://log-it.tech/2017/07/27/configure-kafka-truststore-keystore-using-puppet/ some time ago, and i know that it’s far from bullet proof but it does the job. Now let’s get to the subject, if you want to connect…
-
Implementing logrotate for kafka
Hi, Yes, we will need to implement also logrotate if we want to keep kafka under control. My solution was with puppet, as you probably expected. After i took a look on the documentation related to log4j properties i this i had a configuration figured out that should look like the following erb template #…
-
Securing kafka-manager endpoints with iptables rules behind traefik
Hi, One extra addition to my traefik balancing article from http://log-it.tech/2017/08/19/puppet-implementation-traefik-load-balancer-kafka-manager/ is that even so now we have the balancing capability we still need to restrict access to unsecured endpoint. I thought all the code to be deployable on all of the nodes. If this is taken in consideration, our issue with the firewall rules…
-
Fixing the keystore/trustore distribution code
Hi, There is an extra thing to be added to my article http://log-it.tech/2017/07/27/configure-kafka-truststore-keystore-using-puppet/ As is the code copies the files at each puppet run to the other nodes which not contain the keystore generation code. And to fix this i used yet again another puppet module that should share data between the nodes, you can…
-
Balancing requests to kafka-manager using traefik
Hi, Just wanted to share with you a quite small and simple config to balance the traffic between three machines that have kafka-manager installed. For this i used traefik since it was new to me and i wanted to gain a little bit of experience with it. It’s an interesting solution but it took me…
-
Install Kafka Manager with Puppet
Hi, I will continue on this line with the install of a management tool called Kafka manager using same old Puppet. The main source of the project is here: https://github.com/yahoo/kafka-manager If you scroll down at packaging you will see that you have the possibility to create a .deb package for Ubuntu or Debian setup or…
-
Integrate Kafka with Datadog monitoring using puppet
Hi, Since i was in debt with an article on how to integate Kafka monitoring using Datadog, let me tell you a couple of things about this topic. First of all, we are taking the same config of Kafka with Jolokia that was describe in following article. From the install of the brokers on our…
-
Automigrate data contained in a topic to different Kafka brokers
Hi, Let’s keep it short, you have a cluster composed of three Kafka brokers and you add another two. The details of doing that are pretty straight forward. If they are new, give them a unique broker.id and put the zoookeeper.connect string in order to know on which cluster to register. If they are already…
-
How to deploy Prometheus infrastructure for Kafka monitoring using puppet
Hi, In the last couple of days i worked on deployment of Prometheus server and agent for Kafka monitoring. In that purpose i will share with you the main points that you need to do in order to achieve this. First thing to do is to use the prometheus and grafana modules that you will…
-
Wrong Kafka configuration and deployment using puppet
Hi, I just want to share with you one case that we have last week and that involved some wrong kafka deplyment from puppet that actually filled the filesystems and got our colleagues that were using it for a transport layer on ELK. Lets start with the begining, we have some puppet code to deploy…