kafka puppet

Fixing the kafka-manager puppet code

Hi, we have a new code version for kafka-manager deploy. I will not give more details, just that now it also has a fact for the kafka-password and also some minor changes. Fact looks like this:

require 'facter'
Facter.add(:kafka_manager_pass) do
  setcode do
    if File.exist?(file)
        kafka_manager_pass = Facter::Core::Execution.exec("cat #{file} | grep basicAuthentication.password | cut -d'=' -f2 | tr -d '\"'")
        kafka_manager_pass = "undef"

And also the new puppet code for the class:

class profiles::kafkamanager {

   $zookeeper_connect = hiera('kafkamanager::zookeeperconnect')
   $password = hiera("kafkamanager::password:",'password')
	package {'kafka-manager':
		ensure => installed,
	group { 'kafka-manager':
		ensure => 'present',	
	user { 'kafka-manager':
		ensure => 'present',
		groups => 'kafka-manager'
	Group['kafka-manager'] -> User['kafka-manager']
	file { '/usr/share/kafka-manager' :
    		ensure    => directory,
    		owner     => 'kafka-manager',
    		group      => 'kafka-manager',
    		require     => [ User['kafka-manager'], Group['kafka-manager'], ],
    		recurse    => true,
	file { '/etc/kafka-manager/application.conf':
		ensure => present,
	file_line { 'config_zookeeper':
		path   => '/etc/kafka-manager/application.conf',
		line => "kafka-manager.zkhosts=\"${zookeeper_connect}\"",
		match => 'kafka-manager.zkhosts=\"\"',
    replace => true,
	if ($::kafka_manager_pass == "undef") {
	file_line { 'enable_pass_default':
     path => '/etc/kafka-manager/application.conf',
     match => "basicAuthentication.password=\"password\"",
     line => "basicAuthentication.password=\"${password}\"",
     replace => true,
	elsif ($password != $::kafka_manager_pass) {
  file_line { 'enable_pass':
     path => '/etc/kafka-manager/application.conf',
     match => "basicAuthentication.password=\"${::kafka_manager_pass}\"",
     line => "basicAuthentication.password=\"${password}\"",
     replace => true,
  exec {'restart_kafkamanager':
    command => '/usr/sbin/service kafka-manager restart',
    path => ['/bin','/sbin','/usr/bin','/usr/sbin'],
    refreshonly => true,
    subscribe => File_line['enable_pass'],
    file_line { 'enable_auth':
        path => '/etc/kafka-manager/application.conf',
        match => 'basicAuthentication.enabled=false',
        line => 'basicAuthentication.enabled=true',
        replace => true,
	service { 'kafka-manager':
		ensure => 'running',
		enable => true,
		require => [ File['/usr/share/kafka-manager'], File['/etc/kafka-manager/application.conf'] ],
		subscribe => File["/etc/kafka-manager/application.conf"],

Give it a try!


kafka puppet

Fixing the keystore/trustore distribution code


There is an extra thing to be added to my article

As is the code copies the files at each puppet run to the other nodes which not contain the keystore generation code. And to fix this i used yet again another puppet module that should share data between the nodes, you can find it here

As far as i saw it gets the job done, and in order to use it, you will need to include the following pieces of code to your repo. First of all, one piece of custom fact:

require 'facter'

filename = '/home/kafka/kafka.server.keystore.jks'
Facter.add(:kafkakeystore) do
    setcode do
        if File.file?(filename)
            kafkakeystore = "enabled"
        	kafkakeystore = "disabled"    

If the file is present, this means that the setup is probably activated. For the kafka manifests, if it’s not the node on which the keystore it’s generated we need to share the fact which we actually added in form:

    share_data { "${fqdn}":
      data => [ $::fqdn,$::kafkakeystore ],
      label => 'keystore',

If it’s the node that actually generates and copies the keystore then we will need to include in the class that actually does this kafka_security_gen following piece:

 $data = share_data::retrieve('keystore')
     $data.each |$item| {
   # $servers.each |String $server| {
   if (member($servers,$item[0]) and $item[1] == "disabled") {
        exec{"copy files to ${item[0]}":
            cwd => '/home/kafka',
            path   => '/usr/bin:/usr/sbin:/bin',
            command => "scp /home/kafka/kafka* kafka@${item[0]}:/home/kafka",
            user => 'kafka',

And this should assure you that puppet will not try to copy the keystore on nodes that already has it. Now come to think of it, if you need to refresh the store, it should be a proble, but i will think also for a fix for that and come back.


kafka puppet

Puppet implementation of traefik load balancer for kafka-manager


It’s time to give the puppet implementation for the traefik small case. It is related to the following article

Starting from that i tried to find a puppet module that can actually install the package more or less accurate and i found this

Now, for the service install it works, but for defining of the traefik.toml and rules. toml it was a real pain. First of all one of the function call in the module does not work, and after fixing it, it does’t really align the toml file as required, so i decided to do this in a more simple way. I put the traefik.toml in a file since it doesn’t really contain anything dynamically related to our environment. It looks like:

accessLogsFile = "/var/log/traefik/access.log"
traefikLogsFile = "/var/log/traefik/traefik.log"
logLevel = "DEBUG"
defaultEntryPoints = ["https"]
  address = ":80"
      entryPoint = "https"
  address = ":443"
      CertFile = "/etc/traefik/traefik.crt"
      KeyFile = "/etc/traefik/traefik.key"

address = ":8080"

filename = "/etc/traefik/rules.toml"
watch = true

Now, the config files are stored in /etc/traefik, and i made the convention to store also the self generated certificate for HTTPS also in this location. Sure you can set it dynamically, but for a small load balance and a cluster of a few nodes this should not be a problem.
Ok, as you can see we have a different rules.toml file which in our case it will be created by erb template, and the source is:

      method = "drr"
     <% @kafka_hosts_hash.each do |value, index| %>
    [backends.kafka-manager.servers.server<%= index %>]
    url = "http://<%= value %>:9000"
    weight = 1
    <% end %>
  entrypoints = ["http","https"]
  backend = "kafka-manager"
  passHostHeader = true
  priority = 10

This is pretty straightforward and it will be linked with the last piece of the puzzle, which is the puppet class and it actually looks like this:

class profiles::traefikinstall {
  $version = hiera("profiles::traefik::version",'1.3.5')

  class {'traefik': 
    version           => $version,
  exec {'generate_cert':
  command => "openssl req -newkey rsa:4096 -nodes -sha512 -x509 -days 3650 -nodes -subj \"/CN=${fqdn}/OU=traefik/\" -out /etc/traefik/traefik.crt -keyout /etc/traefik/traefik.key",
  path => ['/usr/bin','/usr/sbin','/bin','/sbin'],
  onlyif => "test ! -f /etc/traefik/traefik.crt"
  } ->
  file {"/etc/traefik/traefik.toml":
    source => 'puppet:///modules/profiles/traefik.toml',
    mode => '0644',
    replace => false,
    notify => Service['traefik'],
  $kafka_hosts = query_nodes("role='kafka'").sort #here it should be any role or fact that indicates that it should have kafka-manager installed
  $kafka_hosts_hash = $ | $index, $value| { [$value,$index+1] }.hash

  file {"/etc/traefik/rules.toml":
    content => template("${module_name}/rules.toml.erb"),
    mode => '0644',
    replace => false,

And this is all the code you need to deploy a traefik instance that it’s “secured” via HTTPS and has load balancing between all kafka-manager instances. Now it’s true that you can secure it by adding iptables rules that restrict traffic on port 9000 (the default kafka manager port) just from the hosts in the cluster, but i will come back also with that part in the future if it will be done.



Install puppet gems on puppet master using hiera


I needed to install a toml-rb gem in order to my puppet traefik module to work and i just want to short post my workaround on doing that automatically. There was some code in our repo for that but it used only hiera array, don’t really know, so i had to write a very short class that can take a hash for the installed process. It looks like this:

class profiles::puppetinstall {
    $packages = hiera_hash('profiles::puppetinstall::packages',undef)
    if packages {

And in my role file called puppetmaster.yaml in this case i had to put:

 - 'profiles::puppetinstall'

      provider: 'puppet_gem'

Now i know that maybe it’s not that elegant, but it fixed my problem. Hopefully i will put all the details related to traefik implementation. And yes, if you are wondering from were can you get the ensure_packages resource, i can tell you it is included in stdlib package

P.S: That was for the puppet agent and standard gems, for the gems that need to be installed on puppet server i needed to write the following piece of code:

$packages_puppetserver = hiera_array('profiles::puppetinstall::puppetserver_packages',undef)
if $packages_puppetserver {
        $packages_puppetserver.each |String $package_name| {
            exec {"install ${package_name}":
                command => "/opt/puppetlabs/bin/puppetserver gem install ${package_name}",
                path => [ '/usr/bin','/usr/sbin','/bin','/sbin' ],
                unless => "/opt/puppetlabs/bin/puppetserver gem list | grep ${package_name}",

The way to put the packages in hiera is similar:

 - 'toml-rb'



Install eyaml module on puppet master


Today i will show how i installed module used for data encrypt in order to safely include it in hiera yaml files)
It really simple as described on The actual step that i couldn’t find explicitly written in the doku and i had to figure it out myself is that you need to modify the config.yaml needed by the module.

  1. gem install hiera-eyaml
  2. puppetserver gem install hiera-eyaml
  3. eyaml createkeys
  4. mv ./keys /etc/puppetlabs/puppet/eyaml
  5. $ chown -R puppet:puppet /etc/puppetlabs/puppet/eyaml
    $ chmod -R 0500 /etc/puppetlabs/puppet/eyaml
    $ chmod 0400 /etc/puppetlabs/puppet/eyaml/*.pem
    $ ls -lha /etc/puppetlabs/puppet/eyaml
    -r——– 1 puppet puppet 1.7K Sep 24 16:24 private_key.pkcs7.pem
    -r——– 1 puppet puppet 1.1K Sep 24 16:24 public_key.pkcs7.pem
  6.  vim /etc/eyaml/config.yaml and add following content:
    pkcs7_private_key: '/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem'
    pkcs7_public_key: '/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem'

If the last step is not executed, you will get the error: [hiera-eyaml-core] No such file or directory – ./keys/public_key.pkcs7.pem

After these configurations you should be able to encrypt files or strings. Short example:

eyaml encrypt -s 'test'
[hiera-eyaml-core] Loaded config from /etc/eyaml/config.yaml
string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAvWHMltzNiYnp0iG6vl6tsgayYimoFQpCFeA8wdE3k6h2OGZAXHLOI+ueEcv+SXVtOsqbP2LxPHe19zJS9cLV4tHu1rUEAW2gstkImI4FoV1/SoPrXNsBBXuoG3j7R4NGPpkhvOQEYIRTT9ssh9hCrzkEMrZ5pZDhS4lNn01Ax1tX99NdmtXaGvTTML/kV061YyN3FaeztSUc01WwpeuHQ+nLouuoVxUUOy/d/5lD5wLKq9t8BYeFG6ekq/D9iGO6D/SNPB0UpVqdCFraAN7rIRNfVDaRbffCSdE59AZr/+atSdUk9cI0oYpG25tHT9x3eWYNNeCLrVAoVMiZ01uR7zA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBHO9P8JfkovKLMdtvaIxAzgBAjiu0/l+Hm+Xaezhp2AWjj]


block: >

Will write something similar for Hiera configuration to use the desired backend.