Getting Data In

How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

Super Champion

one of the customers have a situation whereby there are 1000's of clients with Universal Forwarders in multiple network zones , trying to reach Splunk Heavy Forwarders which are also in multiple network zones. The network zones has to be specific due to security controls, but it is very hard to determine which zone the client (UF) beforehand. As of now, the outputs.conf are hand-crafted manually once the customer identifies which zone the UF is based upon.
I was thinking to push outputs.conf with All Heavy-forwarder-servers in outputs.conf, but I'm sure some of these cannot be reached from the clients. So my question is

  1. How does the UF load-balance behave when it has all (say 10) servers in its outputs.conf list, but only can reach a subset (say 4) of them?
  2. Will it throw error and cause failure on the client? or lot of error logs?
  3. Is there mechanism whereby we can ask the UF not to try the receiver again if it fails N number of times?
0 Karma
1 Solution

Esteemed Legend

It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.

View solution in original post

Esteemed Legend

It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.

View solution in original post

Super Champion

I hope that means, all the data will be intact but will have errors in the UF logs?

0 Karma

Esteemed Legend

No data loss, but possibly data duplication (very unlikely), unless you useAck.

0 Karma

Path Finder

Is this the same for if an indexer has full disk?

0 Karma

Esteemed Legend

Yes, the indexer should put itself into detention/quarantine.

0 Karma

Revered Legend

This documentation page has everything you need to answer you own question.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Protectagainstlossofin-flightdata

Super Champion

I did read this before posting. The actual statement, i wanted to understand from that document was

In all these cases, the forwarder will then attempt to open a connection to the next indexer in the load-balanced group, or to the same indexer again if load-balancing is not enabled.

But I'm not sure whats the impact of having continous non-reachable/timeout indexers

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!