-
Implementing logrotate for kafka
Hi, Yes, we will need to implement also logrotate if we want to keep kafka under control. My solution was with puppet, as you probably expected. After i took a look on the documentation related to log4j properties i this i had a configuration figured out that should look like the following erb template #…
-
Adding custom kafka check consumer.lag to datadog from JMX
Hi, We had the necessity to add the consumer.lag check to datadog. Since we did not have access to the kafka.consumer domain which from what i believe it’s on the client side i decided to connect to the kafka node using JMX (so JConsole was the tool). At MBeans tab you will gladly see that…
-
Securing kafka-manager endpoints with iptables rules behind traefik
Hi, One extra addition to my traefik balancing article from http://log-it.tech/2017/08/19/puppet-implementation-traefik-load-balancer-kafka-manager/ is that even so now we have the balancing capability we still need to restrict access to unsecured endpoint. I thought all the code to be deployable on all of the nodes. If this is taken in consideration, our issue with the firewall rules…
-
Fixing the kafka-manager puppet code
Hi, we have a new code version for kafka-manager deploy. I will not give more details, just that now it also has a fact for the kafka-password and also some minor changes. Fact looks like this: require ‘facter’ Facter.add(:kafka_manager_pass) do setcode do file=’/etc/kafka-manager/application.conf’ if File.exist?(file) kafka_manager_pass = Facter::Core::Execution.exec(“cat #{file} | grep basicAuthentication.password | cut -d’=’…
-
Fixing the keystore/trustore distribution code
Hi, There is an extra thing to be added to my article http://log-it.tech/2017/07/27/configure-kafka-truststore-keystore-using-puppet/ As is the code copies the files at each puppet run to the other nodes which not contain the keystore generation code. And to fix this i used yet again another puppet module that should share data between the nodes, you can…
-
Puppet implementation of traefik load balancer for kafka-manager
Hi, It’s time to give the puppet implementation for the traefik small case. It is related to the following article http://log-it.tech/2017/08/08/balancing-requests-kafka-manager-using-traefik/ Starting from that i tried to find a puppet module that can actually install the package more or less accurate and i found this https://forge.puppet.com/praekeltfoundation/traefik Now, for the service install it works, but for defining of…
-
Balancing requests to kafka-manager using traefik
Hi, Just wanted to share with you a quite small and simple config to balance the traffic between three machines that have kafka-manager installed. For this i used traefik since it was new to me and i wanted to gain a little bit of experience with it. It’s an interesting solution but it took me…
-
Configure kafka truststore and keystore using puppet
Hi, Since the last article was about the template needed to generate the truststore and keystore, now it’s time to give you the rest of the fragments for the deployment with puppet. So, We will have the kafka_security gen class that was called on the last post and it should look like this in the…
-
Generate Kafka key/trust store for TLS activation
Morning folks, Just wanted to share with you also the code for script required on generation of keystore and truststore for Kafka. It is not much but it might be helpful someday. So, the erb script should look like this, and i will show you as well how you can integrate this also with puppet:…
-
Install Kafka Manager with Puppet
Hi, I will continue on this line with the install of a management tool called Kafka manager using same old Puppet. The main source of the project is here: https://github.com/yahoo/kafka-manager If you scroll down at packaging you will see that you have the possibility to create a .deb package for Ubuntu or Debian setup or…