How the heavy forwarder and Splunk DB connect work when indexers is stopping for these upgrading.

[Shuhei052492]

Hi Splunk Professionals,

I am going to upgrade my splunk components.
Along with upgradeing, I am wondering what is the best way to prevent from losing the DB log when stopping indexers.

My enviroment is the below.
- Deploying indexer cluster (3 indexers)
- "Splunk DB connect App(v3.x) is working in Heavy Forwarder. And Heavy Forwarder is monitoring DB logs constantly and forwarding indexers with load balancing.

I am concerned how the heavy forwarder and Splunk DB connect work when indexers is stopping for these upgrading.

In my opinion, I think heavy forwarder will not drop DB logs when stopping indexers , because heavy forwarder is holding wait queues.
While I have no idea about how Splunk DB connect work when wait queues reach the max value, because its inputs setting access DB continuously.

Does anyone have any tips to prevent dropping DB logs while indexer is upgrading?
Or Does anyone know how the Splunk DB connect work when indexers is stopping?
Is there the case that DB logs is dropped cause of Splunk DB connect input setting to send queries?

I will appreciate your any advice and comment.

Best regardes,

[sloshburch]

Splunk generally works like a pipeline. You can submit formally to support for validation, but my understanding is that data stops getting forwarded and when the data out pipeline fill on the HF the mod inputs would stop collecting new data. Or maybe I'm being too optimistic.

[adonio]

1. why would you stop / upgrade all 3 indexers at the exact same time?
2. you can increase your persistent queue on HF
3. if you using a rising column, you can always stop the input and get back to that particular column
4. if you are using batch, stop the input and enable back when indexers are up
5. if you have many inputs, see item #1

[Shuhei052492]

Hi adonio,

I appreciate your much types of tips.
I have both inputs settings, but not much.
So I will let batch setting disabled and set higher value of persistentQueueSize.

I am curious how Splunk DB connect inputs setting work when the persistentQueueSize of forwarder reaches max value.

I have read the doc and understood that forwarder will stop sending data when the persistentQueueSize of forwarder reaches max value.

https://docs.splunk.com/Documentation/Forwarder/7.2.3/Forwarder/Protectagainstthelossofin-flightdata...

However there is no description about input setting.
Does Splunk DB connect input setting (like batch) run and keep requesting query to DB server continuously, even if forwarder stop sending data ?

Regards,

[vishaltaneja070]

@Shuhei052492
Didn't get the point, what is the need to persistent queue in this case. it is better to upgrade indexer one by one. And if multi-site cluster then site by site.

If one indexer is down, the data will load balance to other automatically.