topic Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down? in Getting Data In

How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

koshyk — Tue, 23 May 2017 13:48:55 GMT

one of the customers have a situation whereby there are 1000's of clients with Universal Forwarders in multiple network zones , trying to reach Splunk Heavy Forwarders which are also in multiple network zones. The network zones has to be specific due to security controls, but it is very hard to determine which zone the client (UF) beforehand. As of now, the outputs.conf are hand-crafted manually once the customer identifies which zone the UF is based upon.
I was thinking to push outputs.conf with All Heavy-forwarder-servers in outputs.conf, but I'm sure some of these cannot be reached from the clients. So my question is

How does the UF load-balance behave when it has all (say 10) servers in its outputs.conf list, but only can reach a subset (say 4) of them?
Will it throw error and cause failure on the client? or lot of error logs?
Is there mechanism whereby we can ask the UF not to try the receiver again if it fails N number of times?

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

somesoni2 — Tue, 23 May 2017 13:59:38 GMT

This documentation page has everything you need to answer you own question.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Protectagainstlossofin-flightdata

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

woodcock — Tue, 23 May 2017 14:16:38 GMT

It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

koshyk — Tue, 23 May 2017 14:34:39 GMT

I hope that means, all the data will be intact but will have errors in the UF logs?

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

koshyk — Tue, 23 May 2017 14:37:21 GMT

I did read this before posting. The actual statement, i wanted to understand from that document was

In all these cases, the forwarder will then attempt to open a connection to the next indexer in the load-balanced group, or to the same indexer again if load-balancing is not enabled.

But I'm not sure whats the impact of having continous non-reachable/timeout indexers

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

woodcock — Tue, 23 May 2017 14:54:25 GMT

No data loss, but possibly data duplication (very unlikely), unless you useAck.

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

ridwanahmed — Mon, 11 Nov 2019 18:33:13 GMT

Is this the same for if an indexer has full disk?

Re: How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

woodcock — Tue, 12 Nov 2019 01:31:56 GMT

Yes, the indexer should put itself into detention/quarantine.