Knowledge Management

Why does kvstore lookup record insert fail with error "An error occurred during the last operation ('createCollection', domain: '5', code: '10334')"?

sat94541
Communicator

When user try to update kvstore using command like

| inputlookup [backupdata]  | output 

This errors out with error below

Error :
Could not write to collection [kvstore] error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. 

Preliminary research indicates this could be a database related error.

3-08-2017 08:39:21.322 ERROR KVStorageProvider - An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101
03-08-2017 08:39:21.322 ERROR outputcsv - sid:1488991161.995 Could not write to collection 'vulnerability_management': An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101.
Tags (2)
0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

View solution in original post

abhullar_splunk
Splunk Employee
Splunk Employee

Running Step 6. I was seeing:

error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Basically you are trying to run mongo flat from the file system but the dependencies (like openssl) arent install on the SYSTEM, theyre only installed in SPLUNK.

Either the lib file needs to be installed under default path /usr/lib or the LIB path needs to be set as below
export LD_LIBRARY_PATH=/opt/splunk/lib/:$LD_LIBRARY_PATH

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

immortalraghava
Path Finder

What happens to the backup ? In what scenarios it will be useful ?

0 Karma

sat94541
Communicator

this resolve the issue

0 Karma

mbrunetto
Path Finder

had a similar issue, this fixed it. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...