Knowledge Management

Why does kvstore lookup record insert fail with error "An error occurred during the last operation ('createCollection', domain: '5', code: '10334')"?

Communicator

When user try to update kvstore using command like

| inputlookup [backupdata]  | output 

This errors out with error below

Error :
Could not write to collection [kvstore] error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. 

Preliminary research indicates this could be a database related error.

3-08-2017 08:39:21.322 ERROR KVStorageProvider - An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101
03-08-2017 08:39:21.322 ERROR outputcsv - sid:1488991161.995 Could not write to collection 'vulnerability_management': An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101.
Tags (2)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

View solution in original post

Splunk Employee
Splunk Employee

Running Step 6. I was seeing:

error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Basically you are trying to run mongo flat from the file system but the dependencies (like openssl) arent install on the SYSTEM, theyre only installed in SPLUNK.

Either the lib file needs to be installed under default path /usr/lib or the LIB path needs to be set as below
export LD_LIBRARY_PATH=/opt/splunk/lib/:$LD_LIBRARY_PATH

0 Karma

Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

View solution in original post

Path Finder

What happens to the backup ? In what scenarios it will be useful ?

0 Karma

Communicator

this resolve the issue

0 Karma

Path Finder

had a similar issue, this fixed it. Thanks!

0 Karma