• List differences between two sftp hosts using golang

    Hi,

    Just as a intermediate post as i wanted to play a little bit with golang, let me show you what i managed to put together in some days. I created a virtual machine on which i installed docker and grabbed a sftp image. You can try first two from Docker Hub, it should work.
    So i pulled this image and initiated two containers as shown below:

    eaf3b93798b5        asavartzeth/sftp    "/entrypoint.sh /u..."   21 hours ago        Up About a minute         0.0.0.0:2225->22/tcp   server4
    ec7d7e1d029f        asavartzeth/sftp    "/entrypoint.sh /u..."   21 hours ago        Up About a minute         0.0.0.0:2224->22/tcp   server3
    

    The command to do this looks like:

    docker run --name server3 -v /home/sorin/sftp1:/chroot/sorin:rw -e SFTP_USER=sorin -e SFTP_PASS=pass -p 2224:22 -d asavartzeth/sftp
    docker run --name server4 -v /home/sorin/sftp2:/chroot/sorin:rw -e SFTP_USER=sorin -e SFTP_PASS=pass -p 2225:22 -d asavartzeth/sftp

    Main info to know about these containers is that they should be accessible by user sorin and the path were the external directories are mapped is on /chroot/sorin.

    You can manually test the connection by using a simple command like:

    sftp -P 2224 sorin@localhost

    If you are using the container ip address i observed that you will use the default 22 port to connect to them. Not really clear why but this is not about that.

    Once the servers are up and running you can test the differences between the structure using following code:

    
    package main
    
    import (
    	"fmt"
    
    	"github.com/pkg/sftp"
    	"golang.org/x/crypto/ssh"
    )
    
    type ServerFiles struct {
    	Name  string
    	files []string
    }
    
    func main() {
    
    	server1client := ConnectSftp("localhost:2224", "sorin", "pass")
    	server1files := ReadPath(server1client)
    	server1struct := BuildStruct("172.17.0.2", server1files)
    	server2client := ConnectSftp("localhost:2225", "sorin", "pass")
    	server2files := ReadPath(server2client)
    	server2struct := BuildStruct("172.17.0.3", server2files)
    	diffilesstruct := CompareStruct(server1struct, server2struct)
            for _, values := range diffilestruct.files {
            fmt.Printf("%s %s\n", diffilesstruct.Name, values)
     }
    	CloseConnection(server1client)
    	CloseConnection(server2client)
    }
    func CheckError(err error) {
    	if err != nil {
    		panic(err)
    	}
    }
    func ConnectSftp(address string, user string, password string) *sftp.Client {
    	config := &ssh.ClientConfig{
    		User: user,
    		Auth: []ssh.AuthMethod{
    			ssh.Password(password),
    		},
    		HostKeyCallback: ssh.InsecureIgnoreHostKey(),
    	}
    	conn, err := ssh.Dial("tcp", address, config)
    	CheckError(err)
    
    	client, err := sftp.NewClient(conn)
    	CheckError(err)
    
    	return client
    }
    func ReadPath(client *sftp.Client) []string {
    	var paths []string
    	w := client.Walk("/")
    	for w.Step() {
    		if w.Err() != nil {
    			continue
    		}
    		
    		paths = append(paths, w.Path())
    	}
    	return paths
    }
    func BuildStruct(address string, files []string) *ServerFiles {
    	server := new(ServerFiles)
    	server.Name = address
    	server.files = files
    
    	return server
    }
    func CompareStruct(struct1 *ServerFiles, struct2 *ServerFiles) *ServerFiles {
    
    	diff := difference(struct1.files, struct2.files)
    	diffstruct := new(ServerFiles)
    	for _, value := range diff {
    		for _, valueP := range struct1.files {
    			if valueP == value {
    				
    				diffstruct.Name = struct1.Name
    				diffstruct.files = append(diffstruct.files, valueP)
    			}
    		}
    		for _, valueQ := range struct2.files {
    			if valueQ == value {
    				
    				diffstruct.Name = struct2.Name
    				diffstruct.files = append(diffstruct.files, valueQ)
    			}
    		}
    	}
    	return diffstruct
    }
    func difference(slice1 []string, slice2 []string) []string {
    	var diff []string
    
    	// Loop two times, first to find slice1 strings not in slice2,
    	// second loop to find slice2 strings not in slice1
    	for i := 0; i < 2; i++ {
    		for _, s1 := range slice1 {
    			found := false
    			for _, s2 := range slice2 {
    				if s1 == s2 {
    					found = true
    					break
    				}
    			}
    			// String not found. We add it to return slice
    			if !found {
    				diff = append(diff, s1)
    			}
    		}
    		// Swap the slices, only if it was the first loop
    		if i == 0 {
    			slice1, slice2 = slice2, slice1
    		}
    	}
    
    	return diff
    }
    func CloseConnection(client *sftp.Client) {
    	client.Close()
    }

    This actually connects to each server, reads the hole filepath and puts it on a structure. After this is done for both servers, there is a method that compares only the slice part of the struct and returns the differences. On this differences there is another structure constructed with only the differences.
    It is true that i took the differences func from stackoverflow, and it's far from good code, but i am working on it, this is the first draft, i will post different versions as it gets better.

    The output if there are differences will look like this:

    172.17.0.2 /sorin/subdirectory
    172.17.0.2 /sorin/subdirectory/subtest.file
    172.17.0.2 /sorin/test.file
    172.17.0.3 /sorin/test2
    

    If there are no differences that it will just exit.
    Working on improving my golang experience. Keep you posted.

    Cheers!

  • How to change root password on Debian – after vacation

    Morning,

    Since i had a vacation and completely forgot all my passwords for Debian VM i fixed it using this article. Very useful!

    https://pve.proxmox.com/wiki/Root_Password_Reset

    Cheers!

  • Observer functionality for puppet zookeeper module

    Morning,

    I know it’s been some time since i last posted but i didn’t had the time to play that much. Today i want to share with you the use case in which we needed to modify the module used for the deployment of zookeeper in order to include also observer role.

    The link that describes how this should be activated from version 3.3.0 is located here: https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html

    Taking this situation we are using for deployment module https://github.com/wikimedia/puppet-zookeeper

    It’s not a nice module, trust me, i know, but since we did not want to take the development process from beginning and impact the infrastructure that it’s already deployed we had to cope with this situation by changing what we had.

    Main idea in our case is that since the number of zookeeper members for the election process needs to be 2n+1 in order for the Quorum mechanism to work, deployment of even number of machines was pretty tricky, so to fix this, the extra zookeeper instances over requirements should be set as observers

    A zookeeper observer is a node that it’s not included in the election process and just receives the updates from the cluster.

    My vision is that the best approach for delivery is to activate it in Hiera with a zookeeper::observer parameter per host.

    We can start by including it in the defaults.pp file as follows:

     $observer	      = hiera('zookeeper::observer', false)

    The zoo.conf file deployed for the configuration is being written in the init.pp file so we need to add it also here as parameter

    $observer	   = $::zookeeper::defaults::observer

    Ok, now how do we share the status of each node in the required domain? We will need to use another module https://github.com/WhatsARanjit/puppet-share_data and include in our code something like:

     share_data { $::fqdn:
      	    data  => [ $::fqdn, $observer ],
      	    label => 'role',
        }
       $obsrole = share_data::retrieve('role')
    

    This guarantees us that all servers have and can use the observer flag in the erb template.

    Jumping to the last component of this config, we need to modify the template to have it with the added observer role.

    How do we do that? Basically by rewriting the server information in this format:

    <% if @hosts
     @hosts.sort_by { |name, id| id }.each do |host_id| -%>
    server.<%= host_id[1] %>=<%= host_id[0] %>:2182:2183<% @obsrole.each do |item| if (item[0] == host_id[0]) && item[1] -%>:observer<% end -%><% end -%> 
    <% end -%>
    <% end -%>
    

    Straight forward this compares the values from the two lists and if the flag is true, it adds the observer configuration.
    One last part needs to be added and that is

    <% if @observer == true -%>
    peerType=observer
    <% end -%>
    

    And you are done, if you add zookeeper::observer: true to your yaml file, puppet should rewrite the file and restart Zookeeper service.

    Cheers

  • Memory debug by Heroku guys on Apache Kafka – nice one

    Hi,

    I know, i should write more about my experience with Apache Kafka, have patience, it’s still building, but until then please check this article:

    https://blog.heroku.com/fixing-kafka-memory-leak

    Be aware of the things that you want to include in functionalities and code that is written beside Apache Kafka functionalities, it might get you in to trouble.

    I am very happy that sysdig is used by more and more teams for debug, it’s truly a great tool for this kind of situations.

    Cheers!

  • Docker statistics – way to investigate performance

    Hi,

    I wish it would be mine but it isn’t. Quite good article from this week newsletter related to container stats from Docker containers:

    Analyzing Docker container performance with native tools

    Wish you an enjoyable read.

    Cheers!

  • Kafka limits implementation using puppet

    Morning,

    I keep my promise and provide you with the two simple blocks that are needed to implement limits that we discussed in article http://log-it.tech/2017/10/16/ubuntu-change-ulimit-kafka-not-ignore/

    For the limits module you can use:
    https://forge.puppet.com/puppetlabs/limits

    As for the actual puppet implementation, I took the decision not to restart the service immediately. This being said, it’s dead simple to do it:

    	 file_line {"add_pamd_record":
    	 path => '/etc/pam.d/common-session',
    	 line => 'session required pam_limits.so'
    	 }
    	 limits::fragment {
    	     "*/soft/nofile":
          		value => "100000";
        		"*/hard/nofile":
          		value => "100000";
       		 "kafka/soft/nofile":
          		value => "100000";
        		"kafka/hard/nofile":
          		value => "100000";
      }
    

    This is all you need.

    Cheers

  • Ubuntu – change ulimit for kafka, do not ignore

    Hi,

    Wanna share with you what managed to take me half a day to clarify. I just read in the following article https://docs.confluent.io/current/kafka/deployment.html#file-descriptors-and-mmap
    and learned that in order to optimize kafka, you will need to also change the maximum number of open files. It is nice, but our clusters are deployed on Ubuntu and the images are pretty basic. Not really sure if this is valid for all of the distributions but at least for this one it’s absolutely needed.
    Before trying to setup anything in

    /etc/security/limits.conf

    make sure that you have exported in

    /etc/pam.d/common-session

    line

    session required pam_limits.so

    It is needed in order for ssh, su processes to take the new limits for that user (in our case kafka).
    Doing this will help you define new values on “limits” file. You are now free to setup nofile limit like this for example

    *               soft    nofile          10000
    *		hard	nofile		100000
    kafka		soft 	nofile		10000
    kafka		hard	nofile		100000

    After it is done, you can restart the cluster and check value by finding process with ps-ef | grep kafka and viewing limit file using cat /proc/[kafka-process]/limits.

    I will come back later with also a puppet implementation for this.

    Cheers!

  • Kafka implementation using puppet at IMWorld Bucharest 2017

    Hi,

    I recently had a presentation on how to deploy kafka using puppet and what do you need as a minimum in order to have success in production.
    Here is the presentation:

    Hope it is useful.

    Cheers!

    Update:

    There is also an official version from IMWorld which you can find here:

    And also the article on medium.com that describes it in more technical detail:

    https://medium.com/@sorin.tudor/messaging-kafka-implementation-using-puppet-5438a0ed275d

  • Definitive guide to Kafka, confluent edition

    Hi,

    No technical details today. Just wanted to share with you the Definitive guide to Kafka, book provided by our dear and esteem colleagues from Confluent

    https://www.confluent.io/wp-content/uploads/confluent-kafka-definitive-guide-complete.pdf

    Thank you, it should be an interesting read.

    Cheers!

  • Eyaml hiera configuration for puppet, as promised

    Morning,

    We managed to configure also the hiera backend in order to have eyaml module active. It is related to the following past article http://log-it.tech/2017/05/29/install-eyaml-module-on-puppet-master/. So in the hiera.yaml you bassicaly need to add the following configuration before hierarchy:

    :backends:
      - eyaml
      - yaml
      - puppetdb
    

    and

    :eyaml:
        :datadir: /etc/puppetlabs/hieradata
        :pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
        :pkcs7_public_key:  /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem 
        :extension: 'yaml
    

    at the botton. After this is done, the most essential part is that you created the required symlinks so that the backend is enabled.
    This should be done easily with a bash script like:

    #!/bin/bash
    ln -s /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml
    ln -s /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml_backend.rb /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml_backend.rb
    ln -s /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-eyaml-2.1.0/lib/hiera/backend/eyaml.rb /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/eyaml.rb
    ln -s /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/highline-1.6.21/lib/highline /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/highline/
    ln -s /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/highline-1.6.21/lib/highline.rb /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/highline.rb

    After this is done, it is advised for a puppetdb and puppetserver restart, and you can try testing it by putting a string in hiera and see if a notice prints the required output. Something like

    profiles::test::teststring: '[string generated with eyaml ecrypt -s 'test']'

    and then creating a small class like :

    
    class profiles::test{
    $teststring = hiera('profiles::test::teststring')
    notice {"${teststring}":}
    }

    That should be most of you need in order to do this. Hope it works! 🙂

    Cheers!