Tag: kafka-manager

Fixing the kafka-manager puppet code

Hi, we have a new code version for kafka-manager deploy. I will not give more details, just that now it also has a fact for the kafka-password and also some minor changes. Fact looks like this:

require 'facter'
Facter.add(:kafka_manager_pass) do
  setcode do
    file='/etc/kafka-manager/application.conf'
    if File.exist?(file)
        kafka_manager_pass = Facter::Core::Execution.exec("cat #{file} | grep basicAuthentication.password | cut -d'=' -f2 | tr -d '\"'")
    else
        kafka_manager_pass = "undef"
    end
  end
end

And also the new puppet code for the class:


class profiles::kafkamanager {

   $zookeeper_connect = hiera('kafkamanager::zookeeperconnect')
   $password = hiera("kafkamanager::password:",'password')
	package {'kafka-manager':
		ensure => installed,
	}
	group { 'kafka-manager':
		ensure => 'present',	
	}
	user { 'kafka-manager':
		ensure => 'present',
		groups => 'kafka-manager'
	}
	
	Group['kafka-manager'] -> User['kafka-manager']
	
	file { '/usr/share/kafka-manager' :
    		ensure    => directory,
    		owner     => 'kafka-manager',
    		group      => 'kafka-manager',
    		require     => [ User['kafka-manager'], Group['kafka-manager'], ],
    		recurse    => true,
	}
	file { '/etc/kafka-manager/application.conf':
		ensure => present,
	}
	file_line { 'config_zookeeper':
		path   => '/etc/kafka-manager/application.conf',
		line => "kafka-manager.zkhosts=\"${zookeeper_connect}\"",
		match => 'kafka-manager.zkhosts=\"\"',
    replace => true,
	} 
	if ($::kafka_manager_pass == "undef") {
	file_line { 'enable_pass_default':
     path => '/etc/kafka-manager/application.conf',
     match => "basicAuthentication.password=\"password\"",
     line => "basicAuthentication.password=\"${password}\"",
     replace => true,
       } 
	}
	elsif ($password != $::kafka_manager_pass) {
  file_line { 'enable_pass':
     path => '/etc/kafka-manager/application.conf',
     match => "basicAuthentication.password=\"${::kafka_manager_pass}\"",
     line => "basicAuthentication.password=\"${password}\"",
     replace => true,
       }
  exec {'restart_kafkamanager':
    command => '/usr/sbin/service kafka-manager restart',
    path => ['/bin','/sbin','/usr/bin','/usr/sbin'],
    refreshonly => true,
    subscribe => File_line['enable_pass'],
} 
        }
    file_line { 'enable_auth':
        path => '/etc/kafka-manager/application.conf',
        match => 'basicAuthentication.enabled=false',
        line => 'basicAuthentication.enabled=true',
        replace => true,
        }
           
	service { 'kafka-manager':
		ensure => 'running',
		enable => true,
		require => [ File['/usr/share/kafka-manager'], File['/etc/kafka-manager/application.conf'] ],
		subscribe => File["/etc/kafka-manager/application.conf"],
	}

Give it a try!

Cheers!

August 19, 2017

Puppet implementation of traefik load balancer for kafka-manager

Hi,

It’s time to give the puppet implementation for the traefik small case. It is related to the following article http://log-it.tech/2017/08/08/balancing-requests-kafka-manager-using-traefik/

Starting from that i tried to find a puppet module that can actually install the package more or less accurate and i found this https://forge.puppet.com/praekeltfoundation/traefik

Now, for the service install it works, but for defining of the traefik.toml and rules. toml it was a real pain. First of all one of the function call in the module does not work, and after fixing it, it does’t really align the toml file as required, so i decided to do this in a more simple way. I put the traefik.toml in a file since it doesn’t really contain anything dynamically related to our environment. It looks like:

accessLogsFile = "/var/log/traefik/access.log"
traefikLogsFile = "/var/log/traefik/traefik.log"
logLevel = "DEBUG"
defaultEntryPoints = ["https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/etc/traefik/traefik.crt"
      KeyFile = "/etc/traefik/traefik.key"


[web]
address = ":8080"

[file]
filename = "/etc/traefik/rules.toml"
watch = true

Now, the config files are stored in /etc/traefik, and i made the convention to store also the self generated certificate for HTTPS also in this location. Sure you can set it dynamically, but for a small load balance and a cluster of a few nodes this should not be a problem.
Ok, as you can see we have a different rules.toml file which in our case it will be created by erb template, and the source is:

[backends]
  [backends.kafka-manager]
    [backends.kafka-manager.LoadBalancer]
      method = "drr"
     <% @kafka_hosts_hash.each do |value, index| %>
    [backends.kafka-manager.servers.server<%= index %>]
    url = "http://<%= value %>:9000"
    weight = 1
    <% end %>
[frontends]
  [frontends.kafka-manager]
  entrypoints = ["http","https"]
  backend = "kafka-manager"
  passHostHeader = true
  priority = 10

This is pretty straightforward and it will be linked with the last piece of the puzzle, which is the puppet class and it actually looks like this:

class profiles::traefikinstall {
  $version = hiera("profiles::traefik::version",'1.3.5')

  class {'traefik': 
    version           => $version,
  }
  exec {'generate_cert':
  command => "openssl req -newkey rsa:4096 -nodes -sha512 -x509 -days 3650 -nodes -subj \"/CN=${fqdn}/OU=traefik/O=log-it.tech/L=Bucharest/S=Romania/C=RO\" -out /etc/traefik/traefik.crt -keyout /etc/traefik/traefik.key",
  path => ['/usr/bin','/usr/sbin','/bin','/sbin'],
  onlyif => "test ! -f /etc/traefik/traefik.crt"
  } ->
  file {"/etc/traefik/traefik.toml":
    source => 'puppet:///modules/profiles/traefik.toml',
    mode => '0644',
    replace => false,
    notify => Service['traefik'],
  }
  $kafka_hosts = query_nodes("role='kafka'").sort #here it should be any role or fact that indicates that it should have kafka-manager installed
  $kafka_hosts_hash = $kafka_hosts.map | $index, $value| { [$value,$index+1] }.hash

  file {"/etc/traefik/rules.toml":
    content => template("${module_name}/rules.toml.erb"),
    mode => '0644',
    replace => false,
  }
}

And this is all the code you need to deploy a traefik instance that it’s “secured” via HTTPS and has load balancing between all kafka-manager instances. Now it’s true that you can secure it by adding iptables rules that restrict traffic on port 9000 (the default kafka manager port) just from the hosts in the cluster, but i will come back also with that part in the future if it will be done.

Cheers!

August 19, 2017