Splunk Search

Does anyone know how to monitor all running searches on a search head and their memory usage in real-time?

benjaminlin1019
Explorer

I use one of the S.O.S. queries to get top 20 memory usage queries every 5 minutes, however, it might be easier for us to see that in job inspection. Does anyone know how you can monitor all the running searches and their memory usage in real-time? I would like to see something similar to 'top' results. thanks

yannK
Splunk Employee
Splunk Employee

SOS scripts is a good start.

Otherwise on recent splunk you can use the introspection logs, they run every 30 seconds.
Pick an host (indexer or search-head) and look at the cpu/mem usage per search pid.

index=_introspection host=* source=*/resource_usage.log* | spath | search component=PerProcess                 
    | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval process_class = case( process=="mongod","KV store", process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input") | eval process_class = case( process=="splunkd" AND ((like(args,"-p %start%") AND NOT like(args,"%process-runner%")) OR args=="service"),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class)
| search process_class="search"                     | stats latest(data.pct_cpu) AS resource_usage_cpu_dedup latest(data.mem_used) AS resource_usage_mem_dedup latest(process_class) AS process_class by data.pid,  _time, data.search_props.type,data.search_props.mode,  data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid                   

benjaminlin1019
Explorer

Thanks, I could not find that particular index exist, do you know that any configuration I need to enable first?

0 Karma

yannK
Splunk Employee
Splunk Employee

My bad, it's an underscore index : index=_introspection

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@yannK and @benjaminlin1019 - Search with some performance enhancement. Please check it out if you wish.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
  | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...