9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found

[dbray_sd]

After upgrading Splunk to 9.4.0 and Splunk DB Connect to 3.18.1, all INPUTS have the error:

Checkpoint not found. The input in rising mode is expected to contain a checkpoint.

None of them are pulling in data. Looking over the logs, I see:

2025-01-10 12:16:00.298 +0000 Trace-Id=1d3654ac-86c1-445f-97c6-6919b3f6eb8c [Scheduled-Job-Executor-116] ERROR org.easybatch.core.job.BatchJob - Unable to open record reader
com.splunk.dbx.server.exception.ReadCheckpointFailException: Error(s) occur when reading checkpoint.
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointManager.load(DbInputCheckpointManager.java:71)
at com.splunk.dbx.server.dbinput.task.DbInputTask.loadCheckpoint(DbInputTask.java:133)
at com.splunk.dbx.server.dbinput.recordreader.DbInputRecordReader.executeQuery(DbInputRecordReader.java:82)
at com.splunk.dbx.server.dbinput.recordreader.DbInputRecordReader.open(DbInputRecordReader.java:55)
at org.easybatch.core.job.BatchJob.openReader(BatchJob.java:140)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:97)
at com.splunk.dbx.server.api.service.conf.impl.InputServiceImpl.runTask(InputServiceImpl.java:321)
at com.splunk.dbx.server.api.resource.InputResource.lambda$runInput$1(InputResource.java:183)
at com.splunk.dbx.logging.MdcTaskDecorator.run(MdcTaskDecorator.java:23)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

I'm unable to Edit the config, and update the Check point value. Even thought the Execute Query works, when I try to save the update it gives:

Error(s) occur when reading checkpoint.

Has anybody else successfully upgraded to 9.4.0 and 3.18.1?

[dbray_sd · Accepted solution]

Splunk support came back and stated this is a known issue, and the 9.4.0 update has an issue with the Splunk DB Connect app. The work around was time consuming, but finally everything is back up and running. I had to manually go into:

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf

...and comment out each line with:

tail_rising_column_init_ckpt_value

checkpoint_key

Then restart Splunk, then go into each INPUT config and manually reset the checkpoint value to what was recorded in the tail_rising_column_init_ckpt_value setting. Took forever, but after doing all that and another Splunk restart, only then did all the issues go away. Also noted that the 9.4.0 update removes the legacy tail_rising_column_init_ckpt_value from the db_inputs.conf file, as it is now stored in kvstore, and since kvstore has been updated with 9.4.0 update, that was the overall issue.

Just yet another mess that Splunk updates have caused, but at least support is aware, and they are working hard to properly fix it.

[Meett]

Hello @dbray_sd  Have you tried by cloning older input and creating new one ? Sometimes checkpoint fails during upgrade but cloning new input will create checkpoint and possibly resolves your issue.

[dbray_sd]

Yes, and the issues follows over to the cloned entry.

[kiran_panchavat]

@dbray_sd 

Alright, did you find anything in the internal logs? Are none of the inputs functioning?

Have you taken a backup of the DB Connect add-on before upgrading to the latest version? If you have a backup, please restore it and test again. I haven’t encountered this issue before, but checking the internal logs might provide some insights. If not, it’s best to raise a support ticket.

https://docs.splunk.com/Documentation/DBX/3.18.1/DeployDBX/Troubleshooting 

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.

[dbray_sd]

Thank you, nothing useful in the logs, and I've already opened a ticket. I'll report back.

[kiran_panchavat]

@dbray_sd  Did you perform the health check after upgrading to the latest version of DB Connect?

https://docs.splunk.com/Documentation/DBX/latest/DeployDBX/CheckInstallationHealth 

[dbray_sd]

Thank you @kiran_panchavat  that at least gives me something to investigate further, but also confusing. Health Check is complaining about:

One or more defined connections require the corresponding JDBC driver.

However, those JDBC drivers comes from the Splunk_JDBC_mysql add on app, which I checked and it's running with the latest version. Confusing.