Restoring KV Store Collection: Socket Error or Timeout

[djfangBAH]

Hi everyone!

I'm trying to restore a KV store collection with about 100K records. I am consistently encountering the following error around the 5 minute mark when the restoration process is running:

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout
05-14-2019 10:54:16.741 -0400 WARN  KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backedup_collections_100k.tar.gz", appName="cpz-great-app", collectionName="testingcollection"
05-14-2019 10:54:16.741 -0400 ERROR KVStoreAdminHandler - [ "{ \"ErrorMessage\" : \"Failed to send \\\"update\\\" command with database \\\"s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU\\\": Failed to read 4 bytes: socket error or timeout\" }" ]\n

Based discussions within thread: https://answers.splunk.com/answers/682228/attempting-to-restore-a-kvstore-collection-has-any-1.html

I have also set the following within limits.conf:

[kvstore]
max_documents_per_batch_save = 1000000
max_rows_per_query = 1000000

With it consistently erroring out around the 5 minute mark is leading me to wonder if there's a timeout value within one of the conf settings.

Any thoughts on this would be greatly appreciated!

[DavidHourani]

Hi @sirajnp,

Seems like a network error: "socket error or timeout".

Which Splunk version are you running and is it in a clustered environment ? In case it's an SHC, please make sure that port 8191 is reachable between SHC members.

Cheers,
David

[nagendra1111]

are you able to solve this issue?

[nagendra1111]

@djfangBAH

[sirajnp]

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout

@DavidHourani - We are restoring KVStore from one search head to another, getting above error. What could be the issue