Month: July 2017

  • Configure kafka truststore and keystore using puppet

    Hi,

    Since the last article was about the template needed to generate the truststore and keystore, now it’s time to give you the rest of the fragments for the deployment with puppet.
    So, We will have the kafka_security gen class that was called on the last post and it should look like this in the final form:

    class profiles::kafka_security_gen {
        $pass = hiera('profiles::kafka_security_gen::password','password')
        $keystorepass = hiera('profiles::kafka_security_gen::keystorepass','password')
        
        $cluster_servers = query_nodes("role='kafka'")
        $servers = $cluster_servers.delete("${::fqdn}")
        
        file {'/home/kafka/security.sh':
            owner => kafka,
            group => kafka,
            ensure => file,
            mode => '0755',
            content => template("${module_name}/security.sh.erb"),
            replace => 'no'
    } ->
        exec {'generate_keystore':
        path   => '/usr/bin:/usr/sbin:/bin',
        command => '/home/kafka/security.sh',
        user => 'kafka',
        unless => 'test -f /home/kafka/kafka.server.keystore.jks'
        }
        $servers.each |String $server| {
            exec{"copy files to ${server}":
                cwd => '/home/kafka',
                path   => '/usr/bin:/usr/sbin:/bin',
                command => "scp /home/kafka/kafka* kafka@${server}:/home/kafka",
                user => 'kafka',
            }
            
         }
        ssh_keygen {'kafka':
        home => '/home/kafka/',
        type => rsa,
      }
      @@file {"/home/kafka/.ssh/authorized_keys":
        ensure => present,
        mode => '0600',
        owner => 'kafka',
        group => 'kafka',
        content => "${::sharedkey}",
        tag => "${::instance_tag}",
      }
      
    }

    The only thing i did not manage to fix is the copying of the stores, it will copy them each time, but i will fix that in the near future with some custom facts.
    Ok, now let’s see the integration part with kafka code. Unfortunately, i can not provide you all the code but as far as i can provide you is:

        
    $kafkadirs = ['/home/kafka/', '/home/kafka/.ssh/']
    file { $kafkadirs: 
        ensure => directory,
        mode => '0700',
        owner => 'kafka',
        group => 'kafka'
        }
        $security_enabled = hiera('profiles::kafka::security',false)
    if ($security_enabled == false) {
      $broker_config = {
        ....
      }
      } else {
        # if hostname starts with kafka0 them it's the CA auth (in our case kafka servers are on form kafka0....)
        if ($hostname == 'kafka0') {
        contain 'profiles::kafka_security_gen'
        Class['profiles::kafka_security_gen']
        }
        #this shares the public key on a couple of servers wich are isolated by a tag
        File <<| tag == "${::instance_tag}" |>>
        #path for keystore/truststore
        $keystore_location = '/home/kafka/kafka.server.keystore.jks'
        $truststore_location = '/home/kafka/kafka.server.truststore.jks'
        #broker config
        $broker_config = {
        ....
        'security.inter.broker.protocol' => 'SSL',
        'ssl.client.auth'               => 'none',
        'ssl.enabled.protocols'         => ['TLSv1.2', 'TLSv1.1', 'TLSv1'],
        'ssl.key.password'              => hiera('profiles::kafka_security_gen::password','password'),
        'ssl.keystore.location'         => $keystore_location,
        'ssl.keystore.password'         => hiera('profiles::kafka_security_gen::keystorepass','password'),
        'ssl.keystore.type'             => 'JKS',
        'ssl.protocol'                  => 'TLS',
        'ssl.truststore.location'       => $truststore_location,
        'ssl.truststore.password'       => hiera('profiles::kafka_security_gen::keystorepass','password'),
        'ssl.truststore.type'           => 'JKS',
        'listeners'                     => "PLAINTEXT://${::fqdn}:9092,SSL://${::fqdn}:9093", #both listeners are enabled
        'advertised.listeners'          => "PLAINTEXT://${::fqdn}:9092,SSL://${::fqdn}:9093",
        }
      }
    class { '::kafka::broker':
        config    => $broker_config,
        opts      => hiera('profiles::kafka::jolokia', '-javaagent:/usr/share/java/jolokia-jvm-agent.jar'),
        heap_opts => "-Xmx${jvm_heap_size}M -Xms${jvm_heap_size}M",
      }
    

    This can be easily activated by putting in the hierarchy the profiles::kafka::security option with true and it should do all the work. Once it’s done it can be tested using openssl s_client -debug -connect localhost:9093 -tls1

    The link to the generate article for the security script template is http://log-it.tech/2017/07/23/generate-kafka-keytrust-store-tls-activation/ and also a quite interesting guideline fron Confluence http://docs.confluent.io/current/kafka/ssl.html#configuring-kafka-brokers

    That’s all folks. 🙂

    Cheers

  • Generate Kafka key/trust store for TLS activation

    Morning folks,

    Just wanted to share with you also the code for script required on generation of keystore and truststore for Kafka. It is not much but it might be helpful someday.

    So, the erb script should look like this, and i will show you as well how you can integrate this also with puppet:

    #!/bin/bash
    HOST=<%= @fqdn %>
    PASSWORD=<%= @pass %>
    KEYSTOREPASS=<%= @keystorepass %>
    VALIDITY=365
    
    keytool -keystore kafka.server.keystore.jks -alias $HOST -validity $VALIDITY -genkey -dname "CN=${HOST}, OU=Test, O=Test, L=Bucharest S=Romania C=RO" -storepass $KEYSTOREPASS -keypass $KEYSTOREPASS
    openssl req -new -x509 -keyout ca-key -out ca-cert -days $VALIDITY -subj "/CN=${HOST}/OU=Tests/O=Test/L=Bucharest/S=Romania/C=RO" -passout pass:$PASSWORD
    keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $KEYSTOREPASS -noprompt
    keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $KEYSTOREPASS -noprompt
    keytool -keystore kafka.client.keystore.jks -alias $HOST -validity $VALIDITY -genkey -dname "CN=${HOST}, OU=Test, O=Test, L=Bucharest S=Romania C=RO" -storepass $KEYSTOREPASS -keypass $KEYSTOREPASS
    keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert -storepass $KEYSTOREPASS -noprompt
    keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $KEYSTOREPASS -noprompt
    
    <% @servers.each do |server| -%>
    # <%= server %>
    keytool -keystore kafka.server.keystore.jks -alias <%= server %> -validity $VALIDITY -genkey -dname "CN=<%= server %>, OU=Test, O=Test, L=Bucharest S=Romania C=RO" -storepass $KEYSTOREPASS -keypass $KEYSTOREPASS
    keytool -keystore kafka.server.keystore.jks -alias <%= server %> -certreq -file cert-file-<%= server %>.host -storepass $KEYSTOREPASS
    openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file-<%= server %>.host -out cert-signed-<%= server %>.host -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD
    keytool -keystore kafka.server.keystore.jks -alias <%= server %> -import -file cert-signed-<%= server %>.host -storepass $KEYSTOREPASS -noprompt
    
    keytool -keystore kafka.client.keystore.jks -alias <%= server %> -validity $VALIDITY -genkey -dname "CN=<%= server %>, OU=Test, O=Test, L=Bucharest S=Romania C=RO" -storepass $KEYSTOREPASS -keypass $KEYSTOREPASS
    keytool -keystore kafka.client.keystore.jks -alias <%= server %> -certreq -file cert-file-<%= server %>.client -storepass $KEYSTOREPASS
    openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file-<%= server %>.client -out cert-signed-<%= server %>.client -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD
    keytool -keystore kafka.client.keystore.jks -alias <%= server %> -import -file cert-signed-<%= server %>.client -storepass $KEYSTOREPASS -noprompt
    
    <% end -%>

    As you probably saw already, there are some variables that should be taken from the puppet file, like the list of servers and some passwords for the PEM key and truststore/keystore password.

    Now let’s take a look at the puppet file that should generate this:

    class profiles::kafka_security_gen {
        $pass = hiera('profiles::kafka_security_gen::password','password')
        $keystorepass = hiera('profiles::kafka_security_gen::keystorepass','password')
        
        $cluster_servers = query_nodes("role='kafka'")
        $servers = $cluster_servers.delete("${::fqdn}")
        
        file {'/tmp/security.sh':
            ensure => file,
            mode => '0755',
            content => template("${module_name}/security.sh.erb"),
    }
    }

    This should suffice so far, the idea is that you need to create the truststore and keystore separately for the host that generates them and after that generate and import the CA root certificate, once this is done there are straight forward steps to be taken in order to build the stores.
    I am currently working on a solution to automatic distribute and integrate this on all machines, keep you posted.

    Cheers

  • Jupyter Notebook – very very interesting tool

    Hi,

    As i was taking a look on the Docker newsletter beside Moby and other articles related to that i found this interesting tool and also tutorial/presentation:

    Beside that you can find the official site here: http://jupyter.org

    This caught my attention and i will certainly try this on a machine. I am pretty curios since i believe this is used to power the Wolfram Notebook.

    Cheers!

  • Install Kafka Manager with Puppet

    Hi,

    I will continue on this line with the install of a management tool called Kafka manager using same old Puppet.

    The main source of the project is here:

    https://github.com/yahoo/kafka-manager

    If you scroll down at packaging you will see that you have the possibility to create a .deb package for Ubuntu or Debian setup or a rpm. To do this just clone the project, go into it and run the command

    sbt debian:packageBin
    

    It will take some time but eventually it will create the required package. Now, since you have the package, you can easily ask a friend to upload it to your pipeline apt repo (this is what i did, i don’t have yet the experience on how to do that) and install and configure it via puppet. In order to do that, i “wrote” the following manifest:

    kafkamanager.pp

    class profiles::kafkamanager {
    	
    	$zookeeper_connect = hiera('kafkamanager::zookeeperconnect')
    	package {'kafka-manager':
    		ensure => installed,
    	}
    	group { 'kafka-manager':
    		ensure => 'present',	
    	}
    	user { 'kafka-manager':
    		ensure => 'present',
    		groups => 'kafka-manager'
    	}
    	
    	Group['kafka-manager'] -> User['kafka-manager']
    	
    	file { '/usr/share/kafka-manager' :
        		ensure    => directory,
        		owner     => 'kafka-manager',
        		group      => 'kafka-manager',
        		require     => [ User['kafka-manager'], Group['kafka-manager'], ],
        		recurse    => true,
    	}
    
    	file_line { 'config_zookeeper':
    	    path => '/etc/kafka-manager/application.conf',
    	    match => 'kafka-manager.zkhosts=\"kafka-manager-zookeeper:2181\"',
    	    line => "kafka-manager.zkhosts=\"${zookeeper_connect}\"",
    	    replace => true,
    	}
        
        file_line { 'enable_auth':
            path => '/etc/kafka-manager/application.conf',
            match => 'basicAuthentication.enabled=false',
            line => 'basicAuthentication.enabled=true',
            replace => true,
            }
    	service { 'kafka-manager':
    		ensure => 'running',
    		enable => true,
    		require => [ File['/usr/share/kafka-manager'], File_line['config_zookeeper'] ],	
    	}
    }

    This should be all, you are now free to login with the user and pass that you find in the application.conf file and start mapping Kafka clusters.

    Cheers!