Splunk IT Service Intelligence

Why is IT Service Intelligence (ITSI) kvstore backup timing out?

ssmoot_splunk
Splunk Employee
Splunk Employee

I am trying to backup the kvstore for my IT Service Intelligence deployment and it is failing due to a Splunkd timeout. I see a message in itsi_migration.log:

ERROR [itsi.kvstore.operations] [kvstore_to_json] [] [54560] Failed. Please try running the script again. Error:Error connecting to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_comment: ('The read operation timed out',)

How can I get the backup to complete?

0 Karma
1 Solution

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

View solution in original post

ssmoot_splunk
Splunk Employee
Splunk Employee

Update for Python3, use the following path:

$SPLUNK_HOME/lib/python3.7/site-packages/splunk/rest/__init__.py

0 Karma

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...