Tag: traefik

  • Puppet implementation of traefik load balancer for kafka-manager

    Hi,

    It’s time to give the puppet implementation for the traefik small case. It is related to the following article http://log-it.tech/2017/08/08/balancing-requests-kafka-manager-using-traefik/

    Starting from that i tried to find a puppet module that can actually install the package more or less accurate and i found this https://forge.puppet.com/praekeltfoundation/traefik

    Now, for the service install it works, but for defining of the traefik.toml and rules. toml it was a real pain. First of all one of the function call in the module does not work, and after fixing it, it does’t really align the toml file as required, so i decided to do this in a more simple way. I put the traefik.toml in a file since it doesn’t really contain anything dynamically related to our environment. It looks like:

    accessLogsFile = "/var/log/traefik/access.log"
traefikLogsFile = "/var/log/traefik/traefik.log"
logLevel = "DEBUG"
defaultEntryPoints = ["https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/etc/traefik/traefik.crt"
      KeyFile = "/etc/traefik/traefik.key"


[web]
address = ":8080"

[file]
filename = "/etc/traefik/rules.toml"
watch = true

    Now, the config files are stored in /etc/traefik, and i made the convention to store also the self generated certificate for HTTPS also in this location. Sure you can set it dynamically, but for a small load balance and a cluster of a few nodes this should not be a problem.
    Ok, as you can see we have a different rules.toml file which in our case it will be created by erb template, and the source is:

    [backends]
  [backends.kafka-manager]
    [backends.kafka-manager.LoadBalancer]
      method = "drr"
     <% @kafka_hosts_hash.each do |value, index| %>
    [backends.kafka-manager.servers.server<%= index %>]
    url = "http://<%= value %>:9000"
    weight = 1
    <% end %>
[frontends]
  [frontends.kafka-manager]
  entrypoints = ["http","https"]
  backend = "kafka-manager"
  passHostHeader = true
  priority = 10

    This is pretty straightforward and it will be linked with the last piece of the puzzle, which is the puppet class and it actually looks like this:

    class profiles::traefikinstall {
  $version = hiera("profiles::traefik::version",'1.3.5')

  class {'traefik': 
    version           => $version,
  }
  exec {'generate_cert':
  command => "openssl req -newkey rsa:4096 -nodes -sha512 -x509 -days 3650 -nodes -subj \"/CN=${fqdn}/OU=traefik/O=log-it.tech/L=Bucharest/S=Romania/C=RO\" -out /etc/traefik/traefik.crt -keyout /etc/traefik/traefik.key",
  path => ['/usr/bin','/usr/sbin','/bin','/sbin'],
  onlyif => "test ! -f /etc/traefik/traefik.crt"
  } ->
  file {"/etc/traefik/traefik.toml":
    source => 'puppet:///modules/profiles/traefik.toml',
    mode => '0644',
    replace => false,
    notify => Service['traefik'],
  }
  $kafka_hosts = query_nodes("role='kafka'").sort #here it should be any role or fact that indicates that it should have kafka-manager installed
  $kafka_hosts_hash = $kafka_hosts.map | $index, $value| { [$value,$index+1] }.hash

  file {"/etc/traefik/rules.toml":
    content => template("${module_name}/rules.toml.erb"),
    mode => '0644',
    replace => false,
  }
}

    And this is all the code you need to deploy a traefik instance that it’s “secured” via HTTPS and has load balancing between all kafka-manager instances. Now it’s true that you can secure it by adding iptables rules that restrict traffic on port 9000 (the default kafka manager port) just from the hosts in the cluster, but i will come back also with that part in the future if it will be done.

    Cheers!

  • Balancing requests to kafka-manager using traefik

    Hi,

    Just wanted to share with you a quite small and simple config to balance the traffic between three machines that have kafka-manager installed. For this i used traefik since it was new to me and i wanted to gain a little bit of experience with it.

    It’s an interesting solution but it took me a while to get the pieces working. I will post here my config and will explain the needed part to get it working.

    logLevel = "DEBUG"
defaultEntryPoints = ["http"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
[web]
address = ":8080"

[file]
watch = true

[backends]
  [backends.backend1]
    [backends.backend1.LoadBalancer]
      method = "drr"
    [backends.backend1.servers.server1]
    url = "http://[kafka1.hostname]:9000"
    weight = 1
    [backends.backend1.servers.server2]
    url = "http://[kafka2.hostname]:9000"
    weight = 2
    [backends.backend1.servers.server3]
    url = "http://[kafka3.hostname]:9000"
    weight = 1
[frontends]
  [frontends.frontend1]
  entrypoint = ["http"]
  backend = "backend1"
  passHostHeader = true
  priority = 10

    This is very basic as you can see but it took me a while to understand that you need the file block with watch = true in order for the daemon to see and parse the rules that are listed. You can also have a separate rules file and for that it would be best to consult the traefik documentation.

    I will have to do now the redirect from HTTP to HTTPS in order to secure the connection to frontend. The idea of traefik is that it works like entrypoint -> frontend -> backend and as far as i saw this will be done on the entrypoint level.

    Two extra additions is that you need a default entry point in order for your frontend not to be ignored and also put it on log level DEBUG because otherwise it won’t log much.

    Keep you posted on the progress and also you can find traefik here https://docs.traefik.io

    Cheers!