Knowledge Management

Is it possible to connect directly to MongoDB?

rharrisssi
Path Finder

I want to maintain a lot of data in my KV Store, but in order to do so I have to keep it clean; but aging out old data.

The problem with:

| inputlookup mylookup | where _time>relative_time(now(),"-7d@h") | outputlookup append=false mylookup 

is that it would cause the full database to be replicated again to other search heads and indexers.

Thus I created a script that will issue delete commands when it runs for any records that are too old. However, it can only interact with the API and delete one entry at a time.

If I were able to connect directly to the MongoDB, I could possible issue a "delete from mytable where _time>value" and it would be 1000% more efficient than deleting one record at a time.

Further, I don't think I can delete records fast enough using Python and the API to keep up with what is being added.

Can anyone shed some light on how I can go about connecting directly to the MongoDB?

1 Solution

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

View solution in original post

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

Lucas_K
Motivator

Awesome!

Took me a little while to figure out the conversion from normal lookup search query to mongodb query.

I got it working with the following.

Normal spl based kv lookup query

|inputlookup summary where LastUpdateTime<1468532752

Mongodb query format ( reference : https://docs.mongodb.com/manual/reference/operator/query/lt/ )

{"LastUpdateTime": {"$lt": 1468532752}}

Curl command url encoded ( http://meyerweb.com/eric/tools/dencoder/ )

curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/summary?query=%7B%22LastUpda...

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You cannot use a Mongodb client to connect to Splunk's KVStore. While it is mongodb, its a modified version to fit within the Splunk framework. This isnt supported.

Lucas_K
Motivator

Did you find a resolution to this?

I'm trying to see it I can use dbconnect with mongojdbc and then schedule a search to run the delete.

http://www.unityjdbc.com/mongojdbc/setup/mongodb_jdbc_splunk.pdf

0 Karma

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

0 Karma

ddrillic
Ultra Champion

The mongodb topics page at mongodb

0 Karma
Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...