Knowledge Management

Restoring KV Store Collection: Socket Error or Timeout

djfangBAH
New Member

Hi everyone!

I'm trying to restore a KV store collection with about 100K records. I am consistently encountering the following error around the 5 minute mark when the restoration process is running:

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout
05-14-2019 10:54:16.741 -0400 WARN  KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backedup_collections_100k.tar.gz", appName="cpz-great-app", collectionName="testingcollection"
05-14-2019 10:54:16.741 -0400 ERROR KVStoreAdminHandler - [ "{ \"ErrorMessage\" : \"Failed to send \\\"update\\\" command with database \\\"s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU\\\": Failed to read 4 bytes: socket error or timeout\" }" ]\n

Based discussions within thread: https://answers.splunk.com/answers/682228/attempting-to-restore-a-kvstore-collection-has-any-1.html

I have also set the following within limits.conf:

[kvstore]
max_documents_per_batch_save = 1000000
max_rows_per_query = 1000000

With it consistently erroring out around the 5 minute mark is leading me to wonder if there's a timeout value within one of the conf settings.

Any thoughts on this would be greatly appreciated!

0 Karma

DavidHourani
Super Champion

Hi @sirajnp,

Seems like a network error: "socket error or timeout".

Which Splunk version are you running and is it in a clustered environment ? In case it's an SHC, please make sure that port 8191 is reachable between SHC members.

Cheers,
David

0 Karma

nagendra1111
New Member

are you able to solve this issue?

0 Karma

nagendra1111
New Member

@djfangBAH

0 Karma

sirajnp
Path Finder

05-14-2019 10:54:16.517 -0400 ERROR KVStorageProvider - An error occurred during the last operation ('saveBatchData', domain: '2', code: '4'): Failed to send "update" command with database "s_cpz-coU94dWXKCN+jKROv8LHlXVG8O_testingcollectionBfrJQ4CnAkHyN4DnrvYrGQiU": Failed to read 4 bytes: socket error or timeout

@DavidHourani - We are restoring KVStore from one search head to another, getting above error. What could be the issue

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...