CPU by Search: *NIX

[Ron_Naken]

On a *NIX monolithic search-head/indexer, what is the best method to determine Splunk's CPU load by search at a point in time (no need to calculate avg, etc.)? We are looking to build a report to display something like this:

search index=os sourcetype=ps | multikv     8%     
search index=netapp_perf object=system | ...     5%

Cheers,
Ron

[Ron_Naken · Accepted solution]

Thanks, here's the solution:

index=os sourcetype=ps | multikv | search COMMAND=splunkd | rex field=_raw "search_--id=(?<srid>[^-]+)(?=_)" | search srid=* | join srid [search index=_audit search_id=* search=* | rex field=search_id "'?(?<srid>[^']+)" | fields srid search] | table _time user search pctCPU

Cheers,
Ron

[sloshburch]

If you are using SOS, you can do this search on the SOS rather than the OS index.

I think I may have optimized the performance of this search by moving the 'search srid=*' into a 'search_--id' string in the initial search (thereby reducing the number of events to operate on in the remaining search.

index=os sourcetype=ps search_--id | multikv | search COMMAND=splunkd | rex field=raw "search--id=(?[^-]+)(?=_)" | join srid [search index=_audit search_id=* search=* | rex field=search_id "'?(?[^']+)" | fields srid search] | table _time user search pctCPU

[lguinn2]

Nice answer!

[lguinn2]

If you really wanted to see CPU usage at a point in time, you could collect the data via the *NIX ps command.
Each search in Splunk runs as a separate process. You could even correlate the CPU usage back to the specific search using the search id, which appears in the process information.

The *NIX app has a script that runs the ps command and indexes its output. You just need to enable it - and then write the searches to analyze the data.

Another approach, if you want to look at CPU usage over time: Splunk keeps information about each search that it runs in the _audit index. You can see some of the statistics that can be computed, if you are logged into Splunk as an admin. From the Search app, choose the Status menu (between Search and Views in the menu bar). Then choose "Search Activity" and "Search Details." This dashboard will show the most common and the most expensive searches.