Is it possible to connect directly to MongoDB?

[rharrisssi]

I want to maintain a lot of data in my KV Store, but in order to do so I have to keep it clean; but aging out old data.

The problem with:

| inputlookup mylookup | where _time>relative_time(now(),"-7d@h") | outputlookup append=false mylookup 

is that it would cause the full database to be replicated again to other search heads and indexers.

Thus I created a script that will issue delete commands when it runs for any records that are too old. However, it can only interact with the API and delete one entry at a time.

If I were able to connect directly to the MongoDB, I could possible issue a "delete from mytable where _time>value" and it would be 1000% more efficient than deleting one record at a time.

Further, I don't think I can delete records fast enough using Python and the API to keep up with what is being added.

Can anyone shed some light on how I can go about connecting directly to the MongoDB?

[rharrisssi · Accepted solution]

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

[Lucas_K]

Awesome!

Took me a little while to figure out the conversion from normal lookup search query to mongodb query.

I got it working with the following.

Normal spl based kv lookup query

|inputlookup summary where LastUpdateTime<1468532752

Mongodb query format ( reference : https://docs.mongodb.com/manual/reference/operator/query/lt/ )

{"LastUpdateTime": {"$lt": 1468532752}}

Curl command url encoded ( http://meyerweb.com/eric/tools/dencoder/ )

curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/summary?query=%7B%22LastUpda...

[esix_splunk]

You cannot use a Mongodb client to connect to Splunk's KVStore. While it is mongodb, its a modified version to fit within the Splunk framework. This isnt supported.

[Lucas_K]

Did you find a resolution to this?

I'm trying to see it I can use dbconnect with mongojdbc and then schedule a search to run the delete.

http://www.unityjdbc.com/mongojdbc/setup/mongodb_jdbc_splunk.pdf

[rharrisssi]

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

[ddrillic]

The mongodb topics page at mongodb