Splunk IT Service Intelligence

Why is IT Service Intelligence (ITSI) kvstore backup timing out?

ssmoot_splunk
Splunk Employee
Splunk Employee

I am trying to backup the kvstore for my IT Service Intelligence deployment and it is failing due to a Splunkd timeout. I see a message in itsi_migration.log:

ERROR [itsi.kvstore.operations] [kvstore_to_json] [] [54560] Failed. Please try running the script again. Error:Error connecting to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_comment: ('The read operation timed out',)

How can I get the backup to complete?

0 Karma
1 Solution

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

View solution in original post

ssmoot_splunk
Splunk Employee
Splunk Employee

Update for Python3, use the following path:

$SPLUNK_HOME/lib/python3.7/site-packages/splunk/rest/__init__.py

0 Karma

ssmoot_splunk
Splunk Employee
Splunk Employee

This error is due to the splunkd connection time expiring while waiting for a REST query. To increase the timer, you will have to modify the parameter SPLUNKD_CONNECTION_TIMEOUT in the python restful initialization script.

The script is located:
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/_init_.py

The default for the timer is 30 seconds:
SPLUNKD_CONNECTION_TIMEOUT = 30

I have found that increasing to 300 usually is enough time to allow for the REST command to complete, however, in SHC environments, you might have to increase it even more.

You will need restart splunk to enforce this change.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...