topic Re: How to directly connect to kvstore using a mongodb client and bulk fill my kvstore with 3,000,000 entries? in Knowledge Management

How to directly connect to kvstore using a mongodb client and bulk fill my kvstore with 3,000,000 entries?

bjoernjensen — Thu, 04 Jun 2015 07:06:06 GMT

Hi there,

I would like to initially bulk fill my kvstore with around 3.000.000 entries.

AFAIK the REST API allows to add one entry at a time. Not taking the overhead into account, this takes way too long.

Therefore, I need a direct connection to the kvstore (mongo db 2.6.7-splunk). My first task then would be: "getting a list of all database objects (show dbs)".

I tried using mongo shell 3.0.3. The connection could be established (to the db admin). But I am not allowed to do:

show db
"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0 }",
"code" : 13

But I am connected not using any pem/key file.

With node.js and the package mongodb, I use these options, since I found them in mongod.log:

options: { net: { port: 8191, ssl: { PEMKeyFile: "xxx\etc\auth\server.pem", PEMKeyPassword: "<password>", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "xxx\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "xxx\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } }

But same with this approach:

AssertionError: null == {"name":"MongoError","message":"not authorized on admin to execute command { listDatabases: 1 }","ok":0,"errmsg":"not authorized

My questions so far:

1 How can I bulk fill the kvstore?

2 How can access the kvstore directly?

3 (How) can I use my own mongodb as kvstore?

Thanks - Björn

Re: How to directly connect to kvstore using a mongodb client and bulk fill my kvstore with 3,000,000 entries?

dgladkikh_splun — Mon, 28 Sep 2020 20:11:13 GMT

Two options: a) you can use batch_save REST call http://docs.splunk.com/Documentation/Splunk/6.2.3/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2F.7Bcollection.7D.2Fbatch_save It is limited by default to maximum of 1000 documents, so in your case it will take 3,000 calls.

b) you can create a lookup for this collection, save your data into CSV file and migrate it from CSV into your kvstore collection by issuing a search like "| inputlookup myfile.csv | outputlookup mykvstorecollection_lookup"

2 & 3. That is not supported. Even if you will find out how to make 2 and 3 work - there are will be a huge chance that not everything will work as you expect or something can be broken with following updates. Please let us know if you have some other reasons (other than #1) why would you like to have #2 and #3.

Re: How to directly connect to kvstore using a mongodb client and bulk fill my kvstore with 3,000,000 entries?

bjoernjensen — Thu, 18 Jun 2015 08:48:28 GMT

Thanks for you input on this. I actually missed the batch_save endpoint.

For #2 and #3: One could use the kv store as an interface to master data systems. Moreover in a scenario where (critical) process data is stored within the kv store the customer might be interested in having a non-proprietary access to this kind of data.