How come the CPU utilization of one indexer is always higher than those of our other indexers?

[king2jd]

Hello,

We have a non-clustered indexer environment. We have one indexer (blue line) that is always well above the CPU utilization of the rest. It doesn't appear that there are any more searches on the one in question compared to the others. This indexer was the first indexer we stood up, but I don't see how that could affect anything. The distributed jobs to all indexers is pretty equal across the board. Not sure what is causing the CPU to stay higher than the rest.

Any suggestions?

Thanks!

[gjanders]

There are a few dashboards you could use from Alerts for SplunKAdmins around indexer CPU usage, or github if you prefer.

The main thing to check would be is there a larger volume of data on the said indexer with higher CPU? It's possible the searches use more CPU because there is more data available on that indexer and therefore the searches take longer to execute on that server/take more CPU.
Use the job inspector or isolate down some searches to 1 indexer vs the other or use the queries in one of the indexer CPU dashboards and compare what appears to be taking more CPU per indexer.

There are many other possibilities here but search queries is a common one...

[akocak]

I guess many things can come into picture for question like this:
Are they indexing same data or similar data? any index-time extractions if no HF or UF involvement.
Any OS difference ? Network scans ? Antivirus ? anything that would halt TCP round times
And disk issues ? disk speed tests. here is a search I use when I am looking for indexer issues:

index=_internal sourcetype="splunkd" log_level="ERROR" OR log_level=WARN host=*indexergroup* 
| stats sparkline count dc(host) as uniqhosts last(message) as message values(host) as hosts last(_time) as llast first(_time) as ffirst by punct,component 
| convert ctime(llast) ctime(ffirst) 
| sort -count 
| fields - punct

i have a clustered environment, but this search should be good even for one host