Knowledge Management

Is it possible to connect directly to MongoDB?

Path Finder

I want to maintain a lot of data in my KV Store, but in order to do so I have to keep it clean; but aging out old data.

The problem with:

| inputlookup mylookup | where _time>relative_time(now(),"-7d@h") | outputlookup append=false mylookup 

is that it would cause the full database to be replicated again to other search heads and indexers.

Thus I created a script that will issue delete commands when it runs for any records that are too old. However, it can only interact with the API and delete one entry at a time.

If I were able to connect directly to the MongoDB, I could possible issue a "delete from mytable where _time>value" and it would be 1000% more efficient than deleting one record at a time.

Further, I don't think I can delete records fast enough using Python and the API to keep up with what is being added.

Can anyone shed some light on how I can go about connecting directly to the MongoDB?

1 Solution

Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

View solution in original post

Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

View solution in original post

Motivator

Awesome!

Took me a little while to figure out the conversion from normal lookup search query to mongodb query.

I got it working with the following.

Normal spl based kv lookup query

|inputlookup summary where LastUpdateTime<1468532752

Mongodb query format ( reference : https://docs.mongodb.com/manual/reference/operator/query/lt/ )

{"LastUpdateTime": {"$lt": 1468532752}}

Curl command url encoded ( http://meyerweb.com/eric/tools/dencoder/ )

curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/summary?query=%7B%22LastUpda...

0 Karma

Splunk Employee
Splunk Employee

You cannot use a Mongodb client to connect to Splunk's KVStore. While it is mongodb, its a modified version to fit within the Splunk framework. This isnt supported.

Motivator

Did you find a resolution to this?

I'm trying to see it I can use dbconnect with mongojdbc and then schedule a search to run the delete.

http://www.unityjdbc.com/mongojdbc/setup/mongodb_jdbc_splunk.pdf

0 Karma

Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

0 Karma

Ultra Champion

The mongodb topics page at mongodb

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!