DB Connect not writing new data to indexes because it throws an error while fetching checkpoints

[vmb_splunk]

Under Splunk DB Connect, we have data inputs created which will periodically pull data from our SQLServer database and put it into our indexes.

The SQL queries used have a rising column which acts as a checkpoint column for these inputs.

Things were working fine until KVStore went down on the Splunk server. To bring it up, we followed these steps which worked.

1. Stop the search head that has the stale KV store member.
2. Run the command splunk clean kvstore --local.
3. Restart the search head.
4. Run the command splunk show kvstore-status to verify synchronization.

But, after doing this, Splunk DB Connect stopped writing any new data to the indexes.

The DB Connect logs are filled with error that look like this for all data inputs:

loading checkpoint for input title=CDA_Eportal_JobUserMapping
loading checkpoint by checkpointKey=63863acecd230000be007648 from KV Store for input title=CDA_Eportal_JobUserMapping
error loading checkpoint for input title=CDA_Eportal_JobUserMapping
java.lang.RuntimeException: Can't find checkpoint for DB Input CDA_Eportal_JobUserMapping

I am unable to even edit the data inputs to manually add checkpoints as they fail while saving.

Is there any way to fix all checkpoints or clear all of them so that data gets written to the indexes again? What should I do to fix this issue?

[dimasfs]

Any update on this? We have the same issue here.

[vmb_splunk]

Unfortunately, no.

We used to use Splunk 8.8 when we faced the issue. We thought that since we could not find a solution to this issue we'd upgrade Splunk to 8.9 and DB Connect to its latest version and see if it fixes itself and it did.

So, we just got lucky.