Can you help me with communication and distribution of information from the universal forwarder to Indexer (cluster)?

[efaundez]

Good Morning,

We have the following concern. We currently have several universal forwarders sending information to the indexers, but we see that some servers have outdated information in the outputs.conf.

for example
  
    The current configuration of our cluster is 6 indexer
[tcpout]
disabled = false
defaultGroup = indexCluster

[tcpout: indexCluster]
useACK = true
server = x.x.x.1: 9999, x.x.x.2: 9999, x.x.x.3: 9999, x.x.x.4: 9999, x.x.x.5: 9999, x.x.x.6: 9999

   And certain servers have only some
[tcpout]
disabled = false
defaultGroup = indexCluster

[tcpout: indexCluster]
useACK = true
server = x.x.x.1: 9999, x.x.x.2: 9999, x.x.x.3: 9999, x.x.x.4: 9999

1- Is there any problem if all the machines are not defined in the outputs.conf?

2- We see an overload in some indexer, will it be because all the indexers in our universal forwarder are not defined?

3- When the UF sends information to the cluster, it will be sent by the first IP that establishes communication or the cluster assigns which machine will take this task.

4- What happens when the cluster has a lot of load in an indexer, for example indexer 1 (xxx1: 9999) . Does the cluster perform a balancing and designate another indexer for this task? But if my only forwarder has only that IP pointing, how will i know that the idx2, or idx3 are without less loads, if i do not have these ip defined (xxx2: 9999, xxx3: 9999) in the outputs.conf?

Any information is appreciated

regards

[harsmarvania57 · Accepted solution]

Hi @efaundez,

1. Yes, if you do not define all Indexers servers on all UFs then data load balanced between Indexer servers are not balanced.
2. It might due to this, let's say there are few UFs which has only 4 Indexers in their outputs.conf and they are generating huge amount of data in that case only 4 Indexers will parse those data and due to that there might be possibility that those 4 Indexers are overloaded however remaining 2 Indexer servers have less load compare to other 4.
3. Based on documentation , server will be randomly pick up Indexer server

autoLBFrequency =
* The amount of time, in seconds, that a forwarder sends data to an indexer before redirecting outputs to another indexer in the pool. * Use this setting when you are using automatic load balancing of outputs from universal forwarders (UFs). * Every 'autoLBFrequency' seconds, a new indexer is selected randomly from the list of indexers provided in the server setting of the target group stanza.
* Default: 30
4. If you have only 1 Indexer defined in outputs.conf and if indexer is overloaded and stop receiving data in that case UF will queue data in waitqueue because you are using useACK=true, once wait queue fills up UF will stop sending data to Indexer until Acknowledgement receives back from Indexer . Ref document https://docs.splunk.com/Documentation/Forwarder/7.2.3/Forwarder/Protectagainstthelossofin-flightdata...