Failed ITSI restore from backup...

[hascobot]

Hi,

The very important services_kpi_lookup kvstore got overwritten by a mistake when an operator wrote "|outputlookup services_kpi_lookup" instead of "|inputlookup services_kpi_lookup". This has had extremely big consequences.

The ITSI environment does not work right now. It looks like we have no services, service templates, base searches, etc. Luckily enough the Splunk ITSI keeps backups for a week back by default. However, when we tried to restore to it we got a failed restore. This is our only chance to salvage our environment. It seems like it fails because our ITSI environment is so big. We had over 1000 services and over 8000 KPIs. When we read the logs we see that they say the following:

This last error is the one that we get stuck on right now. The restore from backup functionality seems to not work in our case and we do not know why. Any help would be appreciated. 

Kind Regards,
A Very Concerned Person

[eduncan]

They are timing out because of how many objects are in the KV Store.  Make sure you clean the KV store first and then you can change the default timeout of 12 hours.  The instructions are listed here:https://docs.splunk.com/Documentation/ITSI/4.5.0/Configure/Restore