All Apps and Add-ons

Why DB Connect duplicates events on indexer cluster rolling restart?

joshiro
Communicator

Splunk Enterprise 9.0.1 on premise, clustered search heads and indexers.
DB Connect 3.7.0.

We found out that every time the indexer cluster is restarted, some events are being duplicated in the indexes around the time of the restart.
There are some older threads discussing similar issues but they are using much older versions of the software.

Any ideas on how to troubleshoot/debug/workaround this issue?

Labels (2)
0 Karma
1 Solution

joshiro
Communicator

It turns out that the useACK option in outputs.conf was the source of the problem.
The rolling restart of the indexers doesnt wait for the ACK to be returned to the HF. So the HF will send again the data creating duplicates.

If we were to keep using the ACK then we must stop the HF manually before the rolling restart and start if after it is done.

View solution in original post

0 Karma

joshiro
Communicator

It turns out that the useACK option in outputs.conf was the source of the problem.
The rolling restart of the indexers doesnt wait for the ACK to be returned to the HF. So the HF will send again the data creating duplicates.

If we were to keep using the ACK then we must stop the HF manually before the rolling restart and start if after it is done.

0 Karma

wmazur-splunk
Splunk Employee
Splunk Employee

Db Connect is using KV Store to remember id/timestamp of the latest processed row (whatever was used).

Getting the data from DB, storing it in Splunk and saving the timestamp - it is not an atomic operation so it is possible that KV Store is unavailable when it's needed. 

I would suggest to disable scheduled job via script, do restart, enable job - it will start from the latest processed row (if it's still in the database)

joshiro
Communicator

Thanks for the reply.

The thing is that the HeavyForwarder that contains DB Connect is always online during the rolling restart of the indexers, so unless its using the KV Store of the indexers it shouldnt be unavailable. It might use the KV Store of the indexers for something else but we dont know.

We are going to try to stop the HF before the rolling restart of the indexer and then start it after it finishes.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...