Tag: mongodb

  • Loading unique IP’s in MongoDB

    Hi,

    So today I played a little bit with the possibility of storing the unique IP addresses in a separate table.

    Since I will use a subscription from  ip-api.com, it seems that there is an option to query info by batch processing with a limit of 100 IP’s per payload.

    So, at a first glance there are 227200 unique ip’s in my dataset. That will account for 2272 payloads to be queried.

    The code more or less looks in the following way:

    unique_ip = temp['SourceIP'].unique()
unique_list = [unique_ip[i:i + 100] for i in range(0, len(unique_ip), 100)]
data = []
for i in range(len(unique_list)):
    temp_dict = {}
    temp_dict['id'] = i+1
    temp_dict['payload'] = unique_list[i].tolist()
    data.append(temp_dict)

    Once this is constructed you only need to parse the list element by element and insert it to MongoDB using this code:

    import pymongo

myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["mydatabase"]
mycol = mydb["unique_ip"]
for i in range(len(data)): 
    mycol.insert_one(data[i])

    Next step will involve taking the collection one by one and serve it to the API endpoint.

    Tnx,

    Sorin