Loading unique IP’s in MongoDB

Written by

Sorin Tudor

in

,

Hi,

So today I played a little bit with the possibility of storing the unique IP addresses in a separate table.

Since I will use a subscription from  ip-api.com, it seems that there is an option to query info by batch processing with a limit of 100 IP’s per payload.

So, at a first glance there are 227200 unique ip’s in my dataset. That will account for 2272 payloads to be queried.

The code more or less looks in the following way:

unique_ip = temp['SourceIP'].unique()
unique_list = [unique_ip[i:i + 100] for i in range(0, len(unique_ip), 100)]
data = []
for i in range(len(unique_list)):
    temp_dict = {}
    temp_dict['id'] = i+1
    temp_dict['payload'] = unique_list[i].tolist()
    data.append(temp_dict)

Once this is constructed you only need to parse the list element by element and insert it to MongoDB using this code:

import pymongo

myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["mydatabase"]
mycol = mydb["unique_ip"]
for i in range(len(data)): 
    mycol.insert_one(data[i])

Next step will involve taking the collection one by one and serve it to the API endpoint.

Tnx,

Sorin

More posts