Category: python

  • Prometheus metrics to Pandas data frame

    Hi,

    We are trying to implement a decision tree algorithm in order to see if our resource usage can classify our servers in different categories.

    First step in that process is querying Prometheus from Python and create some data frames with some basic information in order to get them aggregated.

    To that purpose, you can also use the following lines of code:

    import requests
    import copy 
    
    URL = "http://[node_hostname]:9090/api/v1/query?query=metric_to_be_quried[1d]"
      
    r = requests.get(url = URL) 
    
    data = r.json()
    
    data_dict={}
    metric_list = []
    for i in data['data']['result']:
        data_dict = copy.deepcopy(i['metric'])
        for j in i['values']:
            data_dict['time'] = j[0]
            data_dict['value'] = j[1]
            metric_list.append(data_dict)    
    
    df_metric = pd.DataFrame(metric_list)

    Other pieces will follow.

    Cheers

  • Strange problem in puppet run for Ubuntu

    Hi,

    Short sharing of a strange case.

    We’ve written a small manifest in order to distribute some python scripts. You can find the reference here: https://medium.com/metrosystemsro/new-ground-automatic-increase-of-kafka-lvm-on-gcp-311633b0816c

    When you try to run it on Ubuntu 14.04, there is this very strange error:

    Error: Failed to apply catalog: [nil, nil, nil, nil, nil, nil]

    The cause for this is as follows:

    Python 3.4.3 (default, Nov 12 2018, 22:25:49)
    [GCC 4.8.4] on linux (and I believe this is the default max version on trusty)

    In order to install the dependencies, you need python3-pip, so a short search returns following options:

    apt search python3-pip
    Sorting... Done
    Full Text Search... Done
    python3-pip/trusty-updates,now 1.5.4-1ubuntu4 all [installed]
      alternative Python package installer - Python 3 version of the package
    
    python3-pipeline/trusty 0.1.3-3 all
      iterator pipelines for Python 3

    If we want to list all the installed modules with pip3 list, guess what, it’s not working:

    Traceback (most recent call last):
       File "/usr/bin/pip3", line 5, in 
         from pkg_resources import load_entry_point
       File "/usr/local/lib/python3.4/dist-packages/pkg_resources/init.py", line 93, in 
         raise RuntimeError("Python 3.5 or later is required")
     RuntimeError: Python 3.5 or later is required

    So, main conclusion is that it’s not related to puppet, just the incompatibility between version for this old distribution.

    Cheers

  • Small addition for ‘cat’ in Python

    Hi,

    There was a issue on options that aggregate any other ones, like -A for my previous post

    In my view the easiest way to solve it is by storing the options in a tuple.

    Here is the snippet

    run_options = []
    try:
        opts, args = getopt.gnu_getopt(sys.argv[1:-1], 'AbeEnstTv', ['show-all', 'number-nonblank', 'show-ends', 'number', 'show-blank', 'squeeze-blank' 'show-tabs', 'show-nonprinting', 'help', 'version'])
    except getopt.GetoptError:
         print("Something went wrong")
         sys.exit(2)
    for opt, arg in opts:
        if opt in ('-A','--show-all'):
            run_options.append('E')
            run_options.append('T')
        elif opt in ('-b', '--number-nonblank'):
            run_options.append('b')
        elif opt in ('-n', '--number'):
            run_options.append('n')
        elif opt in ('-E', '--show-ends'):
            run_options.append('E')
        elif opt in ('-s', '--squeeze-blank'):
            run_options.append('s')
        elif opt in ('-T', '--show-tabs'):
            run_options.append('T')
     
    final_run_options = tuple(run_options)
    for element in final_run_options:
        if element == 'b':
            content_list = number_nonempty_lines(content_list)
        elif element == 'n':
            content_list = number_all_lines(content_list)   
        elif element == 'E':
            content_list = display_endline(content_list)
        elif element == 's':
            content_list = squeeze_blanks(content_list)
        elif element == 'T':
            content_list = show_tabs(content_list)

    So basically, you store the actual cases in a list which you convert to a tuple to eliminate duplicates. Once you have the final case, you parse it and change the actual content option by option.

    I didn’t have the time to test it but there is no big reason why it should’t work.

    Cheers

  • Linux ‘cat’ in Python – almost complete

    Morning,

    Since I am striving to find useful content to post more often, I took homework for a ‘cat’ written in Python.

    It’s not elegant, and it’s not the best version but it works.

    # -*- coding: utf-8 -*-
    """
    Created on Wed Dec 25 10:28:39 2019
    @author: Sorin
    """
    import sys,getopt,os
    if os.path.isabs(sys.argv[-1:][0]):
        FILENAME= sys.argv[-1:][0]
    else:
        FILENAME = os.getcwd() + "\\" + sys.argv[-1:][0]
    
    def read_content(filename):
        try:
            f = open(filename, "r+")
            content = f.read()
            f.close()
        except IOError as e:
                print("File could not be opened:", e)
                sys.exit(3)
        return content
        
    def transform_content():
        content = read_content(FILENAME)
        content_list = content.split('\n')
        return content_list
    def number_nonempty_lines(content_list):
        i = 0
        for line in content_list:
            if line != '':
                content_list[i] = str(i) + " " + line
            i = i + 1
        return content_list
    def squeeze_blanks(content_list):   
        i = 0
        duplicate_index = []
        for line in content_list:
            if (line == "" or line == "$")  or (str.isdigit(line.split(' ')[0]) and (line.split(' ')[-1] == "" or line.split(' ')[-1] == "$")):
               duplicate_index.append(i+1)
            i = i + 1
        delete_index = []
        for j in range(len(duplicate_index) - 1):
            if  duplicate_index[j] + 1 == duplicate_index[j+1]:
                delete_index.append(duplicate_index[j])
        for element in delete_index:
            content_list.pop(element)
        return content_list    
            
    def number_all_lines(content_list):
        i = 0
        for line in content_list:
            content_list[i] = str(i) + " " + line
            i = i + 1
        return content_list
    
    def display_endline(content_list):
       return [line + "$" for line in content_list]
    
    def show_tabs(content_list):
        print(content_list)
        content_list = [ line.replace('\t','^I') for line in content_list]
        return content_list
    
    content_list =transform_content()
    try:
        opts, args = getopt.gnu_getopt(sys.argv[1:-1], 'AbeEnstTv', ['show-all', 'number-nonblank', 'show-ends', 'number', 'show-blank', 'squeeze-blank' 'show-tabs', 'show-nonprinting', 'help', 'version'])
    except getopt.GetoptError:
         print("Something went wrong")
         sys.exit(2)
    for opt, arg in opts:
        if opt in ('-A','--show-all'):
            content_list = display_endline(content_list)
            content_list = show_tabs(content_list)
        elif opt in ('-b', '--number-nonblank'):
           content_list = number_nonempty_lines(content_list)
        elif opt in ('-n', '--number'):
           content_list = number_all_lines(content_list)
        elif opt in ('-E', '--show-ends'):
            content_list = display_endline(content_list)
        elif opt in ('-s', '--squeeze-blank'):
            content_list = squeeze_blanks(content_list)
        elif opt in ('-T', '--show-tabs'):
            content_list = show_tabs(content_list)
    print('\n'.join(content_list))

    Further improvements will be also posted. I must confess that there are still a couple of things to be fixed, like not running the same options twice, and the issue of putting it to work on very large files, but it will do in this form for now.

    Cheers

  • Automatic increase of Kafka LVM on GCP

    I wrote an article for my company that was published on Medium regarding the topic in the subject. Please see the link

    https://medium.com/metrosystemsro/new-ground-automatic-increase-of-kafka-lvm-on-gcp-311633b0816c

    Thanks

  • Using GCP recommender API for Compute engine

    Let’s keep it short. If you want to use Python libraries for Recommender API, this is how you connect to your project.

    from google.cloud.recommender_v1beta1 import RecommenderClient
    from google.oauth2 import service_account
    def main():
        credential = service_account.Credentials.from_service_account_file('account.json')
        project = "internal-project"
        location = "europe-west1-b"
        recommender = 'google.compute.instance.MachineTypeRecommender'
        client = RecommenderClient(credentials=credential)
        name = client.recommender_path(project, location, recommender)
       
        elements = client.list_recommendations(name,page_size=4)
        for i in elements:
            print(i)
    
    main()

    credential = RecommenderClient.from_service_account_file(‘account.json’) will not return any error, just hang.

    That’s all folks!

  • ELK query using Python with time range

    Short post. Sharing how you make an ELK query from Python using also timestamp:

    es=Elasticsearch([{'host':'[elk_host','port':elk_port}])
    
    query_body_mem = {
        "query": {
            "bool" : {
                "must" : [
                        {
                            "query_string" : {
                            "query": "metricset.module:system metricset.name:memory AND tags:test AND host.name:[hostname]"
                        }
                    },
                    {
                             "range" : {
                                "@timestamp" : {
                                    "gte" : "now-2d",
                                    "lt" :  "now"
                }
            
            }
       
                    }
                ]
            }
       
        }
        
    }
    
    res_mem=es.search(index="metricbeat-*", body=query_body_mem, size=500)
    df_mem = json_normalize(res_mem['hits']['hits'])
    

    And that’s all!

    Cheers

  • Multiple field query in ELK from Python

    Morning,

    There are a lot of pages on how to query ELK stack from Python client library, however, it’s still hard to grab a useful pattern.

    What I wanted is to translate some simple query in Kibana like redis.info.replication.role:master AND beat.hostname:*test AND tags:test into a useful Query DSL JSON.

    It’s worth mentioning that the Python library uses this DSL. Once you have this info, things get much simpler.

    Well, if you search hard enough, you will find a solution, and it should look like.

    another_query_body = {
        "query": {
            "query_string" : {
                "query": "(master) AND (*test) AND (test)",
                "fields": ["redis.info.replication.role", "beat.hostname" , "tags"]
            }
        }
    }

    As you probably guessed, each field maps to a query entry.

    Cheers

  • My introduction to Linear Regression, so far

    Morning,

    Here are some first steps that I want to share with you from my experience with regressions.

    I am new and I took it to step by step, so don’t expect anything new or either very complex.

    First thing, first, we started working on some prediction “algorithms” that should work with data available in the operations domain.

    We needed to have them stored in a centralized location, and it happens that they are sent to ELK. So, the first step is to query then from that location.

    To do that, there is a python client library with a lot of options that I am still beginning to explore. Cutting to the point, to have a regression you need a correlation parameter between the dependent and independent variable, so we thought at first about links between the number of connection and memory usage of a specific service (for example Redis). And this is available with some simple lines of code in Jupyter:

    from elasticsearch import Elasticsearch
    import matplotlib.pyplot as plt
    from pandas.io.json import json_normalize
    es=Elasticsearch([{'host':'ELK_IP','port':'ELK_PORT'}])
    res_redis=es.search(index="metricbeat-redis-*", body={"query": {"match": {'metricset.name': "info" }}}, size=1000)
    df_redis = json_normalize(res_redis['hits']['hits'])
    df_redis_filtered = df_redis[['_source.redis.info.clients.connected','_source.redis.info.memory.used.value']]
    df_redis_filtered['_source.redis.info.memory.used.value'] = df_redis_filtered['_source.redis.info.memory.used.value'] / 10**6
    df_redis_final = df_redis_filtered[df_redis_filtered['_source.redis.info.memory.used.value'] < 300]
    df_redis_final.corr()

    For a little bit of explaining, the used memory needs to be divided to ten to the sixth power in order to transform from bytes to MBytes, and also I wanted to exclude values of memory over 300MB. All good, unfortunately, if you plot the correlation “matrix” between these params, this happens:

    As far as we all should know, a correlation parameter should be as close as possible to 1 or -1, but it’s just not the case.

    And if you want to start plotting, it will look something like:

    So, back to the drawing board, and we now know that we have no clue which columns are correlated. Let us not filter the columns and just remove those that are non-numeric or completely filled with zeros.

    I used this to manipulate the data as simple as possible:

    df_redis_numeric = df_redis.select_dtypes(['number'])
    df_redis_cleaned = df_redis_numeric.loc[:, '_source.redis.info.clients.connected': '_source.redis.info.stats.net.output.bytes' ]
    df_redis_final = df_redis_cleaned.loc[:, (df_redis_cleaned != 0).any(axis=0)]
    df_redis_final.corr()

    And it will bring you a very large matrix with a lot of rows and columns. From that matrix, you can choose two data types that are more strongly correlated. In my example [‘_source.redis.info.cpu.used.user’,’_source.redis.info.cpu.used.sys’]

    If we plot the correlation matrix just for those two colums we are much better than at the start.

    So we are better than before, and we can now start thinking of plotting a regression, and here is the code for that.

    import matplotlib.pyplot as plt
    import numpy as np
    from sklearn import datasets, linear_model
    import pandas as pd
    
    x = df_redis_cpu['_source.redis.info.cpu.used.user']
    y = df_redis_cpu['_source.redis.info.cpu.used.sys']
    
    x = x.values.reshape(-1, 1)
    y = y.values.reshape(-1, 1)
    
    x_train = x[:-250]
    x_test = x[-250:]
    
    y_train = y[:-250]
    y_test = y[-250:]
    
    # Create linear regression object
    regr = linear_model.LinearRegression()
    
    # Train the model using the training sets
    regr.fit(x_train, y_train)
    
    # Plot outputs
    plt.plot(x_test, regr.predict(x_test), color='red',linewidth=3)
    plt.scatter(x_test, y_test,  color='black')
    plt.title('Test Data')
    plt.xlabel('User')
    plt.ylabel('Sys')
    plt.xticks(())
    plt.yticks(())
    
    plt.show()

    Our DataFrame contains 1000 records from which I used 750 to “train” and another 250 to “test”. The output looked this way

    It looks more like a regression, however, what concerns me is the mean square error which is a little bit high.

    So we will need to works further on the DataFrame 🙂

    In order for the linear model to be applied with scikit, the input and output data are transformed into single dimension vectors. If you want to switch back and for example to create a DataFrame from the output of the regression and the actual samples from ELK, it can be done this way:

    data = np.append(np.array(y_test), np.array(y_pred), axis = 1)
    dataset = pd.DataFrame({'test': data[:, 0], 'pred': data[:, 1]})
    dataset['pred'] = dataset['pred'].map(lambda x: '%.2f' % x)

    That is all.

    Cheers!