How can I check the CPU utilization of the SH / indexer from the search?

[damucka]

Hello,

I do not have access to the OS machines of the Splunk but I suspect the CPU bottleneck because my alert jobs are having 3 min lag between scheduling and dispatching. I would like to investigate it further.
Is there any way to query the internal index for the CPU utilization of the SH or indexer?

Kind Regards,
Kamil

[aokur_splunk · Accepted solution]

Another good place to look for bottlenecks are in the monitoring console -> indexing -> performance -> indexing performance:deployment... then look for the queues at 90th percentile and see how much they are utilized... these indexing queues will fill up and cause performance issues, which is a good place to start troubleshooting.

next steps would be to see which sourcetypes are consuming the most resources and optimize them using props/transforms.

additionally alerting is triggered by your search-heads so make sure your sh resources are ok too, you can see them in a similar fashion in the monitoring console under search -> activity