Category: linux

  • Enable time sync on Manjaro

    So I wanted for a while to use and to learn Manjaro and I grabbed Cinnamon 21.1.0

    Installation process is pretty straight forward, I setup the correct time zone and installed all of the default packages.

    Guess what, after rebooting the laptop the timezone was set correctly but the actual time was way off.

    I tried to see if I can easily find a post to explain to me how it’s done but the standard GUI way didn’t work.

    The actual solution is in the code below

    [sorin-20fjs3dr01 ~]# timedatectl
                   Local time: Sb 2021-08-28 13:07:40 EEST
               Universal time: Sb 2021-08-28 10:07:40 UTC
                     RTC time: Sb 2021-08-28 10:07:40
                    Time zone: Europe/Bucharest (EEST, +0300)
    System clock synchronized: no
                  NTP service: inactive
              RTC in local TZ: no
    [sorin-20fjs3dr01 ~]# systemctl status ntpd.service
    ○ ntpd.service - Network Time Service
         Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
         Active: inactive (dead)
    [sorin-20fjs3dr01 ~]#  systemctl status systemd-timesyncd.service
    ○ systemd-timesyncd.service - Network Time Synchronization
         Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: enabled)
         Active: inactive (dead)
           Docs: man:systemd-timesyncd.service(8)
    [sorin-20fjs3dr01 ~]# systemctl start systemd-timesyncd.service
    [sorin-20fjs3dr01 ~]# ^C
    [sorin-20fjs3dr01 ~]# systemctl status systemd-timesyncd.service
    ● systemd-timesyncd.service - Network Time Synchronization
         Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: enabled)
         Active: active (running) since Sat 2021-08-28 13:09:09 EEST; 2h 59min left
           Docs: man:systemd-timesyncd.service(8)
       Main PID: 2080 (systemd-timesyn)
         Status: "Initial synchronization to time server 195.135.194.3:123 (0.manjaro.pool.ntp.org)."
          Tasks: 2 (limit: 19010)
         Memory: 1.3M
            CPU: 51ms
         CGroup: /system.slice/systemd-timesyncd.service
                 └─2080 /usr/lib/systemd/systemd-timesyncd
    
    aug 28 13:09:09 sorin-20fjs3dr01 systemd[1]: Starting Network Time Synchronization...
    aug 28 13:09:09 sorin-20fjs3dr01 systemd[1]: Started Network Time Synchronization.
    aug 28 10:09:10 sorin-20fjs3dr01 systemd-timesyncd[2080]: Initial synchronization to time server 195.135.194.3:123 (0.manjaro.pool.ntp.org).
    [sorin-20fjs3dr01 ~]# systemctl enable systemd-timesyncd.service
    Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /usr/lib/systemd/system/systemd-timesyncd.service.
    Created symlink /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service → /usr/lib/systemd/system/systemd-timesyncd.service.
    [sorin-20fjs3dr01 ~]# 

    Turns out that both ntpd and timesyncd are dead and do not start by default, so the actual fix is by starting and enabling timesyncd.

    Cheers,

    Sorin

  • Recover swap file in vim

    Hi,

    This is a problem that I had since my virtual machine was not stopped properly and my ssh connection was ended prematurely.

    https://superuser.com/questions/204209/how-can-i-recover-the-original-file-from-a-swp-file/205131

    If you have a file.swp and you want to recover it, do as they say. Open the file in VIM and then type recover.

    Cheers

  • Renice until cgroup implementation for process of Yahoo CMAK

    Hi,

    We saw that ex Kafka Manager, now called Yahoo CMAK was using more than enough CPU in some cases, in general related to bad SSL client config.

    It’s not really clear if the CPU usage was real or there was only wait time for resource like memory or I/O (I don’t have an example to post right now, but there are multiple fixes for this).

    The easiest one is to change the nice value for usage. What I observed is that normally it starts with nice value of 0. I guess this is default. General check for this works with

    ps ax -o ni,cmd | grep cmak | grep -v grep

    In order to change this, you can add a crontab line with following command:

    pid=`ps ax -o pid,cmd | grep cmak | grep -v grep |  awk {'print $1'}`; ni=`ps ax -o ni,cmd | grep cmak | grep -v grep |  awk {'print $1'}`; if [ "$ni" = "0" ]; then renice 10 $pid; fi

    Or, even easier than that, add Nice value under [Service] in /etc/systemd/system/multi-user.target.wants/kafka-manager.service

    It does the trick until further cgroup policies are applied.

  • Reset Cinnamon desktop interface

    Hi,

    I recently had an issue with Cinnamon interface, more exactly, my menu panel dissapeared.

    After some quick searches on the net, I found this command:

    gsettings reset-recursively org.cinnamon

    It seems to do the trick.

    Cheers

  • Logs check without ELK 🙂

    Hi,

    We didn’t have the time to implement ELK stack for Kafka logs so if a issue appears it should be done the old fashion way.

    To that purpose, here are two commands that should help you surfing the logs in an easy manner.

    First of all, there is the grep command that should show you the hole line and number.

    A simple example looks like

    grep -nw "2019-06-03" server.log

    This should show you all the lines with date 03.06 from the log of the Kafka broker. The idea is that you can not use it with the standard construct cat server.log | grep -nw “[string]”. It must be used in this specific format.

    Once you found the line number (and it could look just like 95138:java.lang.OutOfMemoryError: Java heap space there is the less command that we can use.

    less +95138 server.log

    And that should give you the line.

    Thanks all folks!

  • Order Linux processes by memory usage

    This one is more for me actually. We have some issues with one puppet instance on which the processes fail, and i wanted to see if there is any way to order them by memory usage.

    So i searched the net and found this link https://unix.stackexchange.com/questions/92493/sorting-down-processes-by-memory-usage

    The command is like

    ps aux --sort -rss | head -10

    And it provides you with following output, at least in my case

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    puppet    6327 70.1 25.5 3585952 1034532 ?     Sl   06:53   7:33 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8778 -Xms1024m -Xmx1024m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
    jenkins   6776  9.6 16.6 4648236 671980 ?      Sl   06:55   0:51 /usr/bin/java -Djava.awt.headless=true -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8780 -Xms1024m -Xmx1024m -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --httpListenAddress=127.0.0.1
    puppetdb  5987 16.8 11.7 3845896 474164 ?      Sl   06:52   2:01 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xmx192m -javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=8779 -cp /opt/puppetlabs/server/apps/puppetdb/puppetdb.jar clojure.main -m puppetlabs.puppetdb.main --config /etc/puppetlabs/puppetdb/conf.d -b /etc/puppetlabs/puppetdb/bootstrap.cfg
    postgres  1458  0.0  2.1 249512 88656 ?        Ss   Nov21   3:10 postgres: checkpointer process                                                                                              
    postgres  6206  0.0  1.4 253448 57984 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb 127.0.0.1(36882) idle                                                                           
    postgres  6209  0.0  0.7 252580 29820 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb 127.0.0.1(36886) idle                                                                           
    postgres  6210  0.0  0.5 254892 22440 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb 127.0.0.1(36888) idle                                                                           
    postgres  6213  0.0  0.5 254320 21416 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb 127.0.0.1(36894) idle                                                                           
    postgres  6205  0.0  0.5 253524 20324 ?        Ss   06:53   0:00 postgres: puppetdb puppetdb 127.0.0.1(36878) idle                       

    As you can probably see, the components are taking slowly but surely more and more memory and since the machine has only 4GB allocated it will probably crash again.

    If this happens, i will manually increase the memory with another 2GB and see where we will go from there.

    Cheers!

  • Golang logging using USER profile on Mint 19

    Hi,

    I committed on learning Golang and as a part of this task i came to play with logging examples. It seems that if you user syslog.LOG_USER the info is stored in the /var/log/syslog.

    Here is the code and also the output

    package main
    import (
    	"io"
    	"log"
    	"log/syslog"
    	"os"
    	"path/filepath"
    )
    func main() {
    	progname := filepath.Base(os.Args[0])
    	sysLog, err := syslog.New(syslog.LOG_INFO|syslog.LOG_USER,progname)
    	if err != nil {
    	log.Fatal(err)
    } else {
    	log.SetOutput(sysLog)
    	}
    	log.Println("LOG_INFO + LOG_USER: Logging in Go!")
    	io.WriteString(os.Stdout,"Will you see this?")
    }
    

    The second line (Will you see this?) is outputed only in console.

    Oct 29 14:30:25 mintworkstation logging[4835]: 2018/10/29 14:30:25 LOG_INFO + LOG_USER: Logging in Go!
    Oct 29 14:30:25 mintworkstation logging[4835]: 2018/10/29 14:30:25 LOG_INFO + LOG_USER: Logging in Go!
    

    P.S.: Managed to find a config file located under /etc/rsyslog.d, called 50-default.conf.
    In this file there is a commented line

    #user.*				-/var/log/user.log
    

    If you uncomment it and restart service with systemctl restart rsyslog, the output will be moved to /var/log/user.log

    Oct 29 14:48:32 mintworkstation NetworkManager[836]:   [1540817312.1683] connectivity: (enp0s31f6) timed out
    Oct 29 14:49:37 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
    Oct 29 14:49:59 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
    Oct 29 14:50:28 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
    Oct 29 14:50:59 mintworkstation logging[5144]: 2018/10/29 14:50:59 LOG_INFO + LOG_USER: Logging in Go!
    Oct 29 14:51:14 mintworkstation gnome-terminal-[2196]: g_menu_insert_item: assertion 'G_IS_MENU_ITEM (item)' failed
    

    Cheers

  • Cgroups management on Linux – first steps

    Hi,

    I didn’t know that much about control groups but i see that there are a big thing in performance and process optimization.
    For the moment i would like to share two important info that i found.
    First, there are three options that you need to activate in you want to play with control group management:

    DefaultCPUAccounting=yes
    DefaultBlockIOAccounting=yes
    DefaultMemoryAccounting=yes
    

    thtat you can find under /etc/systemd/system.conf.

    And, there is also a command that shows CPU utilization along with other info related to the user/system slices – systemd-cgtop.
    If the accounting is not enabled, no details are shown…..once you do that you will have info like this:

    Path                                                                                                                                                                        Tasks   %CPU   Memory  Input/s Output/s
    
    /                                                                                                                                                                              66    9.2        -        -        -
    /user.slice                                                                                                                                                                     -    5.0        -        -        -
    /user.slice/user-1000.slice                                                                                                                                                     -    5.0        -        -        -
    /user.slice/user-1000.slice/session-1.scope                                                                                                                                    47    5.0        -        -        -
    /system.slice                                                                                                                                                                   -    3.8        -        -        -
    /system.slice/lightdm.service                                                                                                                                                   2    3.5        -        -        -
    /system.slice/docker.service                                                                                                                                                    2    0.3        -        -        -
    /system.slice/vboxadd-service.service                                                                                                                                           1    0.0        -        -        -
    /system.slice/ModemManager.service                                                                                                                                              1      -        -        -        -
    /system.slice/NetworkManager.service                                                                                                                                            2      -        -        -        -
    /system.slice/accounts-daemon.service                                                                                                                                           1      -        -        -        -
    /system.slice/acpid.service                                                                                                                                                     1      -        -        -        -
    /system.slice/atd.service                                                                                                                                                       1      -        -        -        -
    /system.slice/avahi-daemon.service                                                                                                                                              2      -        -        -        -
    /system.slice/colord.service                                                                                                                                                    1      -        -        -        -
    /system.slice/cron.service                                                                                                                                                      1      -        -        -        -
    /system.slice/cups-browsed.service                                                                                                                                              1      -        -        -        -
    /system.slice/cups.service                                                                                                                                                      1      -        -        -        -
    /system.slice/dbus.service

    That is all so far. I will let you know once i discover new info.

    Cheers

  • Kernel not compatible with zookeeper version

    Morning,

    It’s important to share this situation with you. This morning i came to the office to see that a cluster that was upgraded/restarted had an issue with Zookeeper instances.

    Symptoms  were clear: instances won’t start completely. But why?

    After a little bit of investigation, i went to the /var/log/syslog (/var/log/zookeeper did not contain any information at all) to see that there is a bad page table in the jvm.

    Java version is:

    java version "1.8.0_111"
    Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
    Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
    

    So, the log showed following lines:

    Aug 16 07:16:04 kafka0 kernel: [  742.349010] init: zookeeper main process ended, respawning
    Aug 16 07:16:04 kafka0 kernel: [  742.925427] java: Corrupted page table at address 7f6a81e5d100
    Aug 16 07:16:05 kafka0 kernel: [  742.926589] PGD 80000000373f4067 PUD b7852067 PMD b1c08067 PTE 80003ffffe17c225
    Aug 16 07:16:05 kafka0 kernel: [  742.928011] Bad pagetable: 000d [#1643] SMP 
    Aug 16 07:16:05 kafka0 kernel: [  742.928011] Modules linked in: dm_crypt serio_raw isofs crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy
    

    Why should the JVM throw a memory error? The main reason is incompatibility with kernel version.

    Let’s take a look in the GRUB config file.

    Looks like we are using for boot:

    menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-baf292e5-0bb6-4e58-8a71-5b912e0f09b6' {
    	recordfail
    	load_video
    	gfxmode $linux_gfx_mode
    	insmod gzio
    	insmod part_msdos
    	insmod ext2
    	if [ x$feature_platform_search_hint = xy ]; then
    	  search --no-floppy --fs-uuid --set=root  baf292e5-0bb6-4e58-8a71-5b912e0f09b6
    	else
    	  search --no-floppy --fs-uuid --set=root baf292e5-0bb6-4e58-8a71-5b912e0f09b6
    	fi
    	linux	/boot/vmlinuz-3.13.0-155-generic root=UUID=baf292e5-0bb6-4e58-8a71-5b912e0f09b6 ro  console=tty1 console=ttyS0
    	initrd	/boot/initrd.img-3.13.0-155-generic
    

    There was also an older version of kernel image available 3.13.0-153.

    Short fix for this is to update the grub.cfg file with the old version and reboot the server.

    Good fix is still in progress. Will post as soon as i have it.

    P.S: I forgot to mention the Zookeeper version:

    Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT

    P.S 2: It seems that the issue is related with the java processes in general not only zookeeper

    Cheers

  • Convert mdx image (daemon tools) to iso

    Hi,

    Some time ago i created some images that were saved in mdx format using (Daemon Tools). Those were Windows times, but i migrated to Debian since then.

    I created a Windows virtual-box machine in order to use them but unfortunately it does not allow to mount them in this format.

    In order to convert them you will have to install a small package called iat using command sudo apt-get install iat . It is found in the default repo.

    One it’s installed , just run iat old_image.mdx old_image.iso

    Cheers