Splunk IT Service Intelligence

Why is IT Service Intelligence (ITSI) kvstore backup timing out?

ssmoot_splunk
Splunk Employee
Splunk Employee

I am trying to backup the kvstore for my IT Service Intelligence deployment and it is failing due to a Splunkd timeout. I see a message in itsi_migration.log:

ERROR [itsi.kvstore.operations] [kvstore_to_json] [] [54560] Failed. Please try running the script again. Error:Error connecting to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_comment: ('The read operation timed out',)

How can I get the backup to complete?

0 Karma
1 Solution

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

View solution in original post

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!