Morning,
I know it’s been some time since i last posted but i didn’t had the time to play that much. Today i want to share with you the use case in which we needed to modify the module used for the deployment of zookeeper in order to include also observer role.
The link that describes how this should be activated from version 3.3.0 is located here: https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html
Taking this situation we are using for deployment module https://github.com/wikimedia/puppet-zookeeper
It’s not a nice module, trust me, i know, but since we did not want to take the development process from beginning and impact the infrastructure that it’s already deployed we had to cope with this situation by changing what we had.
Main idea in our case is that since the number of zookeeper members for the election process needs to be 2n+1 in order for the Quorum mechanism to work, deployment of even number of machines was pretty tricky, so to fix this, the extra zookeeper instances over requirements should be set as observers
A zookeeper observer is a node that it’s not included in the election process and just receives the updates from the cluster.
My vision is that the best approach for delivery is to activate it in Hiera with a zookeeper::observer parameter per host.
We can start by including it in the defaults.pp file as follows:
$observer = hiera('zookeeper::observer', false)
The zoo.conf file deployed for the configuration is being written in the init.pp file so we need to add it also here as parameter
$observer = $::zookeeper::defaults::observer
Ok, now how do we share the status of each node in the required domain? We will need to use another module https://github.com/WhatsARanjit/puppet-share_data and include in our code something like:
share_data { $::fqdn:
data => [ $::fqdn, $observer ],
label => 'role',
}
$obsrole = share_data::retrieve('role')
This guarantees us that all servers have and can use the observer flag in the erb template.
Jumping to the last component of this config, we need to modify the template to have it with the added observer role.
How do we do that? Basically by rewriting the server information in this format:
<% if @hosts
@hosts.sort_by { |name, id| id }.each do |host_id| -%>
server.<%= host_id[1] %>=<%= host_id[0] %>:2182:2183<% @obsrole.each do |item| if (item[0] == host_id[0]) && item[1] -%>:observer<% end -%><% end -%>
<% end -%>
<% end -%>
Straight forward this compares the values from the two lists and if the flag is true, it adds the observer configuration.
One last part needs to be added and that is
<% if @observer == true -%>
peerType=observer
<% end -%>
And you are done, if you add zookeeper::observer: true to your yaml file, puppet should rewrite the file and restart Zookeeper service.
Cheers