Observing consistent uneven load balancing from universal forwarders to indexers

Runals — Thu, 09 Jan 2014 14:12:06 GMT

I've come up with a query to see which indexers our forwarders are sending data to and the results are somewhat eye opening. I'd be interested in other sets of eyes on the logic of my query and to see if others are experiencing the same. The idea is to use the internal logs from the forwarders keying in on the phrase "connected to idx". The query is

index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | stats count by indexer | sort indexer

I'm sorting the indexers in numeric order (IP) because that is the way they are listed in the outputs.conf. What is interesting is the visual profile of the distribution is basically the same regardless of the time of day or time span of the query. In other words my first indexer receives about 2x the connections as the others whether that is in the last 5 minutes or 7 days. The same visual profile regardless of time/time range is generally true for the others. I've come up with a query to match the results of this data to the GB indexed per indexer in the same time period but it needs to be run in the deployment monitor app - wasn't going to rewrite all the macros. I also had to do some heavy massaging in order to match IP to indexer name and then to sort correctly. You might not have that issue /shrug. Run it in its base form and see (I've left out post append stuffs)

index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | eval "Splunk Server" = indexer | stats count by "Splunk Server" | append [search `all_indexers_summary`]  | sort "Splunk Server"

Environment

20 indexers

1300 forwarders

Using auto load balancing built into outputs.conf

Forwarder versions range from 4.3.4 to 6.0.1

Re: Observing consistent uneven load balancing from universal forwarders to indexers

MuS — Thu, 09 Jan 2014 15:03:26 GMT

Hi Runals,

see this answer from @lguinn, maybe this helps ....

cheers, MuS

topic Re: Observing consistent uneven load balancing from universal forwarders to indexers in Getting Data In

Observing consistent uneven load balancing from universal forwarders to indexers

Re: Observing consistent uneven load balancing from universal forwarders to indexers