Knowledge Management

Restoring KV Store Collection: Socket Error or Timeout

djfangBAH
New Member

Hi everyone!

I'm trying to restore a KV store collection with about 100K records. I am consistently encountering the following error around the 5 minute mark when the restoration process is running:

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout
05-14-2019 10:54:16.741 -0400 WARN  KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backedup_collections_100k.tar.gz", appName="cpz-great-app", collectionName="testingcollection"
05-14-2019 10:54:16.741 -0400 ERROR KVStoreAdminHandler - [ "{ \"ErrorMessage\" : \"Failed to send \\\"update\\\" command with database \\\"s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU\\\": Failed to read 4 bytes: socket error or timeout\" }" ]\n

Based discussions within thread: https://answers.splunk.com/answers/682228/attempting-to-restore-a-kvstore-collection-has-any-1.html

I have also set the following within limits.conf:

[kvstore]
max_documents_per_batch_save = 1000000
max_rows_per_query = 1000000

With it consistently erroring out around the 5 minute mark is leading me to wonder if there's a timeout value within one of the conf settings.

Any thoughts on this would be greatly appreciated!

0 Karma

DavidHourani
Super Champion

Hi @sirajnp,

Seems like a network error: "socket error or timeout".

Which Splunk version are you running and is it in a clustered environment ? In case it's an SHC, please make sure that port 8191 is reachable between SHC members.

Cheers,
David

0 Karma

nagendra1111
New Member

are you able to solve this issue?

0 Karma

nagendra1111
New Member

@djfangBAH

0 Karma

sirajnp
Path Finder

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout

@DavidHourani - We are restoring KVStore from one search head to another, getting above error. What could be the issue

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...