Getting Data In

How to determine what causes the unevenness of the indexing rate across the indexers?

danielbb
Motivator

The indexer with the highest indexing rate has a double indexing rate than the lowest indexer, based on the Monitoring Console.
How can see this rate for the indexers for a month or so? and hopefully with the hosts that each one gets data from.

Tags (2)
0 Karma
1 Solution

adonio
Ultra Champion

there are plenty of things to check, below are the first steps i will suggest

start here, this will tell you how many events are indexed and how many unique hosts are sending data to each indexer
| tstats count as event_count dc(host) as u_host where index=* by splunk_server
if you have very uneven numbers there, start looking at outputs.conf and verify your hosts have the appropriate outputs.conf configurations
you can also start by checking load over time:
| tstats count as event_count where index=* by splunk_server _time span=1d | timechart span=1d max(event_count_ as total_events by splunk_server

hope it leads you in the right direction

View solution in original post

adonio
Ultra Champion

there are plenty of things to check, below are the first steps i will suggest

start here, this will tell you how many events are indexed and how many unique hosts are sending data to each indexer
| tstats count as event_count dc(host) as u_host where index=* by splunk_server
if you have very uneven numbers there, start looking at outputs.conf and verify your hosts have the appropriate outputs.conf configurations
you can also start by checking load over time:
| tstats count as event_count where index=* by splunk_server _time span=1d | timechart span=1d max(event_count_ as total_events by splunk_server

hope it leads you in the right direction

danielbb
Motivator

Thank you @adonio.

When running - | tstats count as event_count by splunk_server _time span=1d | timechart span=1d max(event_count) as total_events by splunk_server we see the following -

alt text

How come some of the cells are empty? These indexers were up and running every day...

0 Karma

adonio
Ultra Champion

forgot to add the where clause to the tstats see my fixed above
also, if there is no data / events that day for that index, then its 0 / null

danielbb
Motivator

Really really interesting @adonio - adding the where clause changed everything.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...