one of the customers have a situation whereby there are 1000's of clients with Universal Forwarders in multiple network zones , trying to reach Splunk Heavy Forwarders which are also in multiple network zones. The network zones has to be specific due to security controls, but it is very hard to determine which zone the client (UF) beforehand. As of now, the outputs.conf are hand-crafted manually once the customer identifies which zone the UF is based upon.
I was thinking to push outputs.conf with All Heavy-forwarder-servers in outputs.conf, but I'm sure some of these cannot be reached from the clients. So my question is
It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.
It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.
I hope that means, all the data will be intact but will have errors in the UF logs?
No data loss, but possibly data duplication (very unlikely), unless you useAck
.
Is this the same for if an indexer has full disk?
Yes, the indexer should put itself into detention/quarantine.
This documentation page has everything you need to answer you own question.
https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Protectagainstlossofin-flightdata
I did read this before posting. The actual statement, i wanted to understand from that document was
In all these cases, the forwarder will then attempt to open a connection to the next indexer in the load-balanced group, or to the same indexer again if load-balancing is not enabled.
But I'm not sure whats the impact of having continous non-reachable/timeout indexers