Why does kvstore lookup record insert fail with error "An error occurred during the last operation ('createCollection', domain: '5', code: '10334')"?

[sat94541]

When user try to update kvstore using command like

| inputlookup [backupdata]  | output 

This errors out with error below

Error :
Could not write to collection [kvstore] error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. 

Preliminary research indicates this could be a database related error.

3-08-2017 08:39:21.322 ERROR KVStorageProvider - An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101
03-08-2017 08:39:21.322 ERROR outputcsv - sid:1488991161.995 Could not write to collection 'vulnerability_management': An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101.

[abhullar_splunk]

Running Step 6. I was seeing:

error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Basically you are trying to run mongo flat from the file system but the dependencies (like openssl) arent install on the SYSTEM, theyre only installed in SPLUNK.

Either the lib file needs to be installed under default path /usr/lib or the LIB path needs to be set as below
export LD_LIBRARY_PATH=/opt/splunk/lib/:$LD_LIBRARY_PATH

[rbal_splunk · Accepted solution]

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

[immortalraghava]

What happens to the backup ? In what scenarios it will be useful ?

[sat94541]

this resolve the issue

[mbrunetto]

had a similar issue, this fixed it. Thanks!