Why DB Connect duplicates events on indexer cluster rolling restart?

[joshiro]

Splunk Enterprise 9.0.1 on premise, clustered search heads and indexers.
DB Connect 3.7.0.

We found out that every time the indexer cluster is restarted, some events are being duplicated in the indexes around the time of the restart.
There are some older threads discussing similar issues but they are using much older versions of the software.

Any ideas on how to troubleshoot/debug/workaround this issue?

[joshiro · Accepted solution]

It turns out that the useACK option in outputs.conf was the source of the problem.
The rolling restart of the indexers doesnt wait for the ACK to be returned to the HF. So the HF will send again the data creating duplicates.

If we were to keep using the ACK then we must stop the HF manually before the rolling restart and start if after it is done.

[wmazur-splunk]

Db Connect is using KV Store to remember id/timestamp of the latest processed row (whatever was used).

Getting the data from DB, storing it in Splunk and saving the timestamp - it is not an atomic operation so it is possible that KV Store is unavailable when it's needed. 

I would suggest to disable scheduled job via script, do restart, enable job - it will start from the latest processed row (if it's still in the database)

[joshiro]

Thanks for the reply.

The thing is that the HeavyForwarder that contains DB Connect is always online during the rolling restart of the indexers, so unless its using the KV Store of the indexers it shouldnt be unavailable. It might use the KV Store of the indexers for something else but we dont know.

We are going to try to stop the HF before the rolling restart of the indexer and then start it after it finishes.