How to connect Splunk to MongoDB, or is there a KV store bulk delete option?

[mathiask]

We consider using the Splunk KV store as backend for our application.
Besides some operational benefits, we would not have to run a separate database.

But since we process, i.e. insert and delete many entries, we need some kind of efficient or bulk delete operation which the REST API currently does not support. Side note, we are on Python 3.
https://answers.splunk.com/answers/465340/how-to-perform-a-bulkbatch-delete-of-kv-store-reco.html
If you choose to fix this, because apparently MongoDB should be able to do this, please do so.

Sadly, we slowly moved to the decision that we have to run our own MongoDB.
At least we would like to integrate it.
What is the best way to do it?
DBConnect does not support it: http://docs.splunk.com/Documentation/DBX/2.4.0/DeployDBX/Supporteddatabases
Hunk App for MongoDB is likely outdated, 6.1 and Hunk is now part of Splunk Enterprise https://splunkbase.splunk.com/app/1810/#/overview

Cheers

[dschmidt_cfi]

I have never tried SCL (Software Collections) to downgrade a version like from Python 3.x to 2.7.x; however it should work. The only change would be the command line. Another option is to have Splunk run the command using the Python version it has installed, Under 6.5.2 it is reporting 2.7.11.

/opt/splunk/bin/splunk cmd python --version

[mathiask]

Thanks for your input.

We have a message processing engine in place, which is written in python 3.
We want to use this engine to interact with the KV store, like inserting, updating some lookup data.
Whether there is a python2 env on the system or Splunk or not, does not really help.

At the moment we access the KV store directly through the REST API.
Massive deletions are still an issue.

[goodsellt]

Yea it seems that if you're using Python3 you are screwed. I'm using the splunk-sdk to do KVStore manipulation and bulk deletes work pretty well on it, I can do ~10K deletes in a couple seconds (if you're willing to some kind of Python2 compatibility in your app). However, one thing to note is that for bulk deletes to work well with the Splunk implementation, you have to be able to define the records you want with a WHERE clause (ex you need to filter them the same way you would do a search that goes '|inputlookup mykvstore WHERE XXXX' returns those records). Then you have to shoot that query in it's JSON form (see https://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore) to the endpoint.

See my code in this answers page for my working example (https://answers.splunk.com/answers/474643/how-do-i-filter-data-with-pythonrest-kv-store-quer.html ). If you can't use the Python2 code, you should be able to do the same using the REST API's storage/collections/data/{collection} method and passing a query as a request param, see that RESTkvstore reference as it should be in there. It mimics the query functionality.