How to determine what causes the unevenness of the indexing rate across the indexers?

[danielbb]

The indexer with the highest indexing rate has a double indexing rate than the lowest indexer, based on the Monitoring Console.
How can see this rate for the indexers for a month or so? and hopefully with the hosts that each one gets data from.

[adonio · Accepted solution]

there are plenty of things to check, below are the first steps i will suggest

start here, this will tell you how many events are indexed and how many unique hosts are sending data to each indexer
| tstats count as event_count dc(host) as u_host where index=* by splunk_server
if you have very uneven numbers there, start looking at outputs.conf and verify your hosts have the appropriate outputs.conf configurations
you can also start by checking load over time:
| tstats count as event_count where index=* by splunk_server _time span=1d | timechart span=1d max(event_count_ as total_events by splunk_server

hope it leads you in the right direction

[danielbb]

Thank you @adonio.

When running - | tstats count as event_count by splunk_server _time span=1d | timechart span=1d max(event_count) as total_events by splunk_server we see the following -

How come some of the cells are empty? These indexers were up and running every day...

[adonio]

forgot to add the where clause to the tstats see my fixed above
also, if there is no data / events that day for that index, then its 0 / null

[danielbb]

Really really interesting @adonio - adding the where clause changed everything.