-
Using Google Gemini to optimise my scraping code
I wanted for Gemini to take a look on the code that I posted here Here is his answer: It lacks a couple of lines like: And But otherwise it works like charm and it’s faster than my first implementation Use chatbots, they know better most of the times. Cheers!
-
Scraping Alcopa auction site using python
Just for reference a small example of scrapping https://www.alcopa-auction.fr/ for new cars on specific brand. It can be improved a lot but just as a reference of the code. As for the code description, here is the one from Github Copilot, you judge if it’s good or not.
-
Using line_profiler in Python
Currently I am reading one of the chapters from Python Data Science Handbook and saw that there are some magic commands that can be used in IPython. One of them gives you the possibility to time the execution and it’s called %timeit. Going further it seems that there are other commands like %prun but it…
-
Merging initial data load with IP info
Getting back to my traffic statistics project, I managed to play with aggregation of the two main collections that had the information I needed. First of all there is the normal code to load data from Mongo and check also the columns that can be used for the join. Which is available below After this,…
-
Loading unique IP’s in MongoDB
Hi, So today I played a little bit with the possibility of storing the unique IP addresses in a separate table. Since I will use a subscription from ip-api.com, it seems that there is an option to query info by batch processing with a limit of 100 IP’s per payload. So, at a first glance…
-
Loading data to a Mongo database for further processing
Hi, Since I needed my data to be available for further processing in a centralized manner, I have decided to store the first draft as well as further queries to location API in a Mongo database. Here is the short code snippet that was used for this task: From the concatenated file, my interest is…
-
Microsoft Teams blocked by pfBlockerNG
Hi, One short tip to remember. I’ve been struggling for a while now with the fact that pfBlockerNG was blocking my Teams connection for whatever reason. I couldn’t understand what was the correct way to fix this until today. I should have known that there isn’t a range of IPs that can be whitelisted to…
-
Traffic statistics – new project
Hi, For some time I wanted to understand how the traffic on my networking is actually shaped. To that purpose, at first I purchased a Synology router but it seems that it hasn’t that much traffic logging capabilities, so I kept it and put in front of it the following box. It’s a cool toy…
-
Exclusive SASL on Zookeeper connections
Something related to following article. It seems that even if SASL is configured until version 3.6.1, Zookeeper will still allow anonymous connections and actions. There is now a new configuration available that will restrict such events and you can find it documented on the official Apache Zookeeper administration guide (zookeeper.sessionRequireClientSASLAuth) The main catch is that…
-
Datadog and GCP are “friends” up to a point
Hi, Since in the last period I preferred to publish more on Medium, let me give you the link to the latest article. There is an interesting case in which the combination of automation, Goggle Cloud Platform and Datadog didn’t go as we expected. https://medium.com/metrosystemsro/puppet-datadog-google-cloud-platform-recipe-for-a-small-outage-310166e551f1 Hope you enjoy! I will get back with more also…