-
Plotting unique values in a dataframe column
Today, I began experimenting with a Kaggle dataframe for the first time. Here is the link to it: https://www.kaggle.com/datasets/open-powerlifting/powerlifting-database To make a long story short, if you want to plot the count for each individual column and capture the unique values of a particular column inside that dataframe, you can simply achieve it by using the…
-
Getting unique IP from mongoDB and use it for traceroute query
Hi, Some time ago, I tried to write some python code in order to grab each unique IP from my traffic logs and trying to trace it so that We can find similar nodes which were used for the jumps. This is also a good exercise in order to improve the basic dataframe information and…
-
Fixing “local issuer certificate” in Python
I am putting this here more as a reference since this error appeared a couple of time when using urllib library. For example there is the classical case of And if you run it for the first time in your Jupyter Kernel, it will return The easiest way to fix it is by adding two…
-
Using Google Gemini to optimise my scraping code
I wanted for Gemini to take a look on the code that I posted here Here is his answer: It lacks a couple of lines like: And But otherwise it works like charm and it’s faster than my first implementation Use chatbots, they know better most of the times. Cheers!
-
Scraping Alcopa auction site using python
Just for reference a small example of scrapping https://www.alcopa-auction.fr/ for new cars on specific brand. It can be improved a lot but just as a reference of the code. As for the code description, here is the one from Github Copilot, you judge if it’s good or not.
-
Using line_profiler in Python
Currently I am reading one of the chapters from Python Data Science Handbook and saw that there are some magic commands that can be used in IPython. One of them gives you the possibility to time the execution and it’s called %timeit. Going further it seems that there are other commands like %prun but it…
-
Merging initial data load with IP info
Getting back to my traffic statistics project, I managed to play with aggregation of the two main collections that had the information I needed. First of all there is the normal code to load data from Mongo and check also the columns that can be used for the join. Which is available below After this,…
-
Loading unique IP’s in MongoDB
Hi, So today I played a little bit with the possibility of storing the unique IP addresses in a separate table. Since I will use a subscription from ip-api.com, it seems that there is an option to query info by batch processing with a limit of 100 IP’s per payload. So, at a first glance…
-
Loading data to a Mongo database for further processing
Hi, Since I needed my data to be available for further processing in a centralized manner, I have decided to store the first draft as well as further queries to location API in a Mongo database. Here is the short code snippet that was used for this task: From the concatenated file, my interest is…
-
Start of the traffic project
So, I managed to gather about 1 GB of records from the pfsense installation and grab them from the box (filter.log files that you can find under /var/log). And I have a list of 16 logs that I need to concatenate. I had a lot of trouble concatenating it since I tried multiple times to…