Splunk ITSI

Why is IT Service Intelligence (ITSI) kvstore backup timing out?

ssmoot_splunk
Splunk Employee
Splunk Employee

I am trying to backup the kvstore for my IT Service Intelligence deployment and it is failing due to a Splunkd timeout. I see a message in itsi_migration.log:

ERROR [itsi.kvstore.operations] [kvstore_to_json] [] [54560] Failed. Please try running the script again. Error:Error connecting to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_comment: ('The read operation timed out',)

How can I get the backup to complete?

0 Karma
1 Solution

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

View solution in original post

ssmoot_splunk
Splunk Employee
Splunk Employee

Update for Python3, use the following path:

$SPLUNK_HOME/lib/python3.7/site-packages/splunk/rest/__init__.py

0 Karma

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...