Does anyone know how to monitor all running searches on a search head and their memory usage in real-time?

[benjaminlin1019]

I use one of the S.O.S. queries to get top 20 memory usage queries every 5 minutes, however, it might be easier for us to see that in job inspection. Does anyone know how you can monitor all the running searches and their memory usage in real-time? I would like to see something similar to 'top' results. thanks

[yannK]

SOS scripts is a good start.

Otherwise on recent splunk you can use the introspection logs, they run every 30 seconds.
Pick an host (indexer or search-head) and look at the cpu/mem usage per search pid.

index=_introspection host=* source=*/resource_usage.log* | spath | search component=PerProcess                 
    | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval process_class = case( process=="mongod","KV store", process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input") | eval process_class = case( process=="splunkd" AND ((like(args,"-p %start%") AND NOT like(args,"%process-runner%")) OR args=="service"),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class)
| search process_class="search"                     | stats latest(data.pct_cpu) AS resource_usage_cpu_dedup latest(data.mem_used) AS resource_usage_mem_dedup latest(process_class) AS process_class by data.pid,  _time, data.search_props.type,data.search_props.mode,  data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid                   

[benjaminlin1019]

Thanks, I could not find that particular index exist, do you know that any configuration I need to enable first?

[yannK]

My bad, it's an underscore index : index=_introspection

[VatsalJagani]

@yannK and @benjaminlin1019 - Search with some performance enhancement. Please check it out if you wish.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
  | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid