Why is IT Service Intelligence (ITSI) kvstore backup timing out?

[ssmoot_splunk]

I am trying to backup the kvstore for my IT Service Intelligence deployment and it is failing due to a Splunkd timeout. I see a message in itsi_migration.log:

ERROR [itsi.kvstore.operations] [kvstore_to_json] [] [54560] Failed. Please try running the script again. Error:Error connecting to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_comment: ('The read operation timed out',)

How can I get the backup to complete?

[ssmoot_splunk]

Update for Python3, use the following path:

$SPLUNK_HOME/lib/python3.7/site-packages/splunk/rest/__init__.py

[ssmoot_splunk · Accepted solution]

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.