Enable time sync on Manjaro

So I wanted for a while to use and to learn Manjaro and I grabbed Cinnamon 21.1.0

Installation process is pretty straight forward, I setup the correct time zone and installed all of the default packages.

Guess what, after rebooting the laptop the timezone was set correctly but the actual time was way off.

I tried to see if I can easily find a post to explain to me how it’s done but the standard GUI way didn’t work.

The actual solution is in the code below

[sorin-20fjs3dr01 ~]# timedatectl
               Local time: Sb 2021-08-28 13:07:40 EEST
           Universal time: Sb 2021-08-28 10:07:40 UTC
                 RTC time: Sb 2021-08-28 10:07:40
                Time zone: Europe/Bucharest (EEST, +0300)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
[sorin-20fjs3dr01 ~]# systemctl status ntpd.service
○ ntpd.service - Network Time Service
     Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
[sorin-20fjs3dr01 ~]#  systemctl status systemd-timesyncd.service
○ systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-timesyncd.service(8)
[sorin-20fjs3dr01 ~]# systemctl start systemd-timesyncd.service
[sorin-20fjs3dr01 ~]# ^C
[sorin-20fjs3dr01 ~]# systemctl status systemd-timesyncd.service
● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-08-28 13:09:09 EEST; 2h 59min left
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 2080 (systemd-timesyn)
     Status: "Initial synchronization to time server ("
      Tasks: 2 (limit: 19010)
     Memory: 1.3M
        CPU: 51ms
     CGroup: /system.slice/systemd-timesyncd.service
             └─2080 /usr/lib/systemd/systemd-timesyncd

aug 28 13:09:09 sorin-20fjs3dr01 systemd[1]: Starting Network Time Synchronization...
aug 28 13:09:09 sorin-20fjs3dr01 systemd[1]: Started Network Time Synchronization.
aug 28 10:09:10 sorin-20fjs3dr01 systemd-timesyncd[2080]: Initial synchronization to time server (
[sorin-20fjs3dr01 ~]# systemctl enable systemd-timesyncd.service
Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /usr/lib/systemd/system/systemd-timesyncd.service.
Created symlink /etc/systemd/system/ → /usr/lib/systemd/system/systemd-timesyncd.service.
[sorin-20fjs3dr01 ~]# 

Turns out that both ntpd and timesyncd are dead and do not start by default, so the actual fix is by starting and enabling timesyncd.




Recover swap file in vim


This is a problem that I had since my virtual machine was not stopped properly and my ssh connection was ended prematurely.

If you have a file.swp and you want to recover it, do as they say. Open the file in VIM and then type recover.



Renice until cgroup implementation for process of Yahoo CMAK


We saw that ex Kafka Manager, now called Yahoo CMAK was using more than enough CPU in some cases, in general related to bad SSL client config.

It’s not really clear if the CPU usage was real or there was only wait time for resource like memory or I/O (I don’t have an example to post right now, but there are multiple fixes for this).

The easiest one is to change the nice value for usage. What I observed is that normally it starts with nice value of 0. I guess this is default. General check for this works with

ps ax -o ni,cmd | grep cmak | grep -v grep

In order to change this, you can add a crontab line with following command:

pid=`ps ax -o pid,cmd | grep cmak | grep -v grep |  awk {'print $1'}`; ni=`ps ax -o ni,cmd | grep cmak | grep -v grep |  awk {'print $1'}`; if [ "$ni" = "0" ]; then renice 10 $pid; fi

Or, even easier than that, add Nice value under [Service] in /etc/systemd/system/

It does the trick until further cgroup policies are applied.


Reset Cinnamon desktop interface


I recently had an issue with Cinnamon interface, more exactly, my menu panel dissapeared.

After some quick searches on the net, I found this command:

gsettings reset-recursively org.cinnamon

It seems to do the trick.



Logs check without ELK :)


We didn’t have the time to implement ELK stack for Kafka logs so if a issue appears it should be done the old fashion way.

To that purpose, here are two commands that should help you surfing the logs in an easy manner.

First of all, there is the grep command that should show you the hole line and number.

A simple example looks like

grep -nw "2019-06-03" server.log

This should show you all the lines with date 03.06 from the log of the Kafka broker. The idea is that you can not use it with the standard construct cat server.log | grep -nw “[string]”. It must be used in this specific format.

Once you found the line number (and it could look just like 95138:java.lang.OutOfMemoryError: Java heap space there is the less command that we can use.

less +95138 server.log

And that should give you the line.

Thanks all folks!

linux puppet

Order Linux processes by memory usage

This one is more for me actually. We have some issues with one puppet instance on which the processes fail, and i wanted to see if there is any way to order them by memory usage.

So i searched the net and found this link

The command is like

ps aux --sort -rss | head -10

And it provides you with following output, at least in my case

puppet    6327 70.1 25.5 3585952 1034532 ?     Sl   06:53   7:33 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -XX:OnOutOfMemoryError=kill -9 %p -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8778 -Xms1024m -Xmx1024m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
jenkins   6776  9.6 16.6 4648236 671980 ?      Sl   06:55   0:51 /usr/bin/java -Djava.awt.headless=true -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8780 -Xms1024m -Xmx1024m -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --httpListenAddress=
puppetdb  5987 16.8 11.7 3845896 474164 ?      Sl   06:52   2:01 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8779 -cp /opt/puppetlabs/server/apps/puppetdb/puppetdb.jar clojure.main -m puppetlabs.puppetdb.main --config /etc/puppetlabs/puppetdb/conf.d -b /etc/puppetlabs/puppetdb/bootstrap.cfg
postgres  1458  0.0  2.1 249512 88656 ?        Ss   Nov21   3:10 postgres: checkpointer process                                                                                              
postgres  6206  0.0  1.4 253448 57984 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb idle                                                                           
postgres  6209  0.0  0.7 252580 29820 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb idle                                                                           
postgres  6210  0.0  0.5 254892 22440 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb idle                                                                           
postgres  6213  0.0  0.5 254320 21416 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb idle                                                                           
postgres  6205  0.0  0.5 253524 20324 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb idle                       

As you can probably see, the components are taking slowly but surely more and more memory and since the machine has only 4GB allocated it will probably crash again.

If this happens, i will manually increase the memory with another 2GB and see where we will go from there.


golang linux

Golang logging using USER profile on Mint 19


I committed on learning Golang and as a part of this task i came to play with logging examples. It seems that if you user syslog.LOG_USER the info is stored in the /var/log/syslog.

Here is the code and also the output

package main
import (
func main() {
	progname := filepath.Base(os.Args[0])
	sysLog, err := syslog.New(syslog.LOG_INFO|syslog.LOG_USER,progname)
	if err != nil {
} else {
	log.Println("LOG_INFO + LOG_USER: Logging in Go!")
	io.WriteString(os.Stdout,"Will you see this?")

The second line (Will you see this?) is outputed only in console.

Oct 29 14:30:25 mintworkstation logging[4835]: 2018/10/29 14:30:25 LOG_INFO + LOG_USER: Logging in Go!
Oct 29 14:30:25 mintworkstation logging[4835]: 2018/10/29 14:30:25 LOG_INFO + LOG_USER: Logging in Go!

P.S.: Managed to find a config file located under /etc/rsyslog.d, called 50-default.conf.
In this file there is a commented line

#user.*				-/var/log/user.log

If you uncomment it and restart service with systemctl restart rsyslog, the output will be moved to /var/log/user.log

Oct 29 14:48:32 mintworkstation NetworkManager[836]:   [1540817312.1683] connectivity: (enp0s31f6) timed out
Oct 29 14:49:37 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
Oct 29 14:49:59 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
Oct 29 14:50:28 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
Oct 29 14:50:59 mintworkstation logging[5144]: 2018/10/29 14:50:59 LOG_INFO + LOG_USER: Logging in Go!
Oct 29 14:51:14 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed



Cgroups management on Linux – first steps


I didn’t know that much about control groups but i see that there are a big thing in performance and process optimization.
For the moment i would like to share two important info that i found.
First, there are three options that you need to activate in you want to play with control group management:


thtat you can find under /etc/systemd/system.conf.

And, there is also a command that shows CPU utilization along with other info related to the user/system slices – systemd-cgtop.
If the accounting is not enabled, no details are shown…..once you do that you will have info like this:

Path                                                                                                                                                                        Tasks   %CPU   Memory  Input/s Output/s

/                                                                                                                                                                              66    9.2        -        -        -
/user.slice                                                                                                                                                                     -    5.0        -        -        -
/user.slice/user-1000.slice                                                                                                                                                     -    5.0        -        -        -
/user.slice/user-1000.slice/session-1.scope                                                                                                                                    47    5.0        -        -        -
/system.slice                                                                                                                                                                   -    3.8        -        -        -
/system.slice/lightdm.service                                                                                                                                                   2    3.5        -        -        -
/system.slice/docker.service                                                                                                                                                    2    0.3        -        -        -
/system.slice/vboxadd-service.service                                                                                                                                           1    0.0        -        -        -
/system.slice/ModemManager.service                                                                                                                                              1      -        -        -        -
/system.slice/NetworkManager.service                                                                                                                                            2      -        -        -        -
/system.slice/accounts-daemon.service                                                                                                                                           1      -        -        -        -
/system.slice/acpid.service                                                                                                                                                     1      -        -        -        -
/system.slice/atd.service                                                                                                                                                       1      -        -        -        -
/system.slice/avahi-daemon.service                                                                                                                                              2      -        -        -        -
/system.slice/colord.service                                                                                                                                                    1      -        -        -        -
/system.slice/cron.service                                                                                                                                                      1      -        -        -        -
/system.slice/cups-browsed.service                                                                                                                                              1      -        -        -        -
/system.slice/cups.service                                                                                                                                                      1      -        -        -        -

That is all so far. I will let you know once i discover new info.


kafka linux

Kernel not compatible with zookeeper version


It’s important to share this situation with you. This morning i came to the office to see that a cluster that was upgraded/restarted had an issue with Zookeeper instances.

Symptoms  were clear: instances won’t start completely. But why?

After a little bit of investigation, i went to the /var/log/syslog (/var/log/zookeeper did not contain any information at all) to see that there is a bad page table in the jvm.

Java version is:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

So, the log showed following lines:

Aug 16 07:16:04 kafka0 kernel: [  742.349010] init: zookeeper main process ended, respawning
Aug 16 07:16:04 kafka0 kernel: [  742.925427] java: Corrupted page table at address 7f6a81e5d100
Aug 16 07:16:05 kafka0 kernel: [  742.926589] PGD 80000000373f4067 PUD b7852067 PMD b1c08067 PTE 80003ffffe17c225
Aug 16 07:16:05 kafka0 kernel: [  742.928011] Bad pagetable: 000d [#1643] SMP 
Aug 16 07:16:05 kafka0 kernel: [  742.928011] Modules linked in: dm_crypt serio_raw isofs crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy

Why should the JVM throw a memory error? The main reason is incompatibility with kernel version.

Let’s take a look in the GRUB config file.

Looks like we are using for boot:

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-baf292e5-0bb6-4e58-8a71-5b912e0f09b6' {
	gfxmode $linux_gfx_mode
	insmod gzio
	insmod part_msdos
	insmod ext2
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  baf292e5-0bb6-4e58-8a71-5b912e0f09b6
	  search --no-floppy --fs-uuid --set=root baf292e5-0bb6-4e58-8a71-5b912e0f09b6
	linux	/boot/vmlinuz-3.13.0-155-generic root=UUID=baf292e5-0bb6-4e58-8a71-5b912e0f09b6 ro  console=tty1 console=ttyS0
	initrd	/boot/initrd.img-3.13.0-155-generic

There was also an older version of kernel image available 3.13.0-153.

Short fix for this is to update the grub.cfg file with the old version and reboot the server.

Good fix is still in progress. Will post as soon as i have it.

P.S: I forgot to mention the Zookeeper version:

Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT

P.S 2: It seems that the issue is related with the java processes in general not only zookeeper



Convert mdx image (daemon tools) to iso


Some time ago i created some images that were saved in mdx format using (Daemon Tools). Those were Windows times, but i migrated to Debian since then.

I created a Windows virtual-box machine in order to use them but unfortunately it does not allow to mount them in this format.

In order to convert them you will have to install a small package called iat using command sudo apt-get install iat . It is found in the default repo.

One it’s installed , just run iat old_image.mdx old_image.iso