Splunk Search

Does anyone know how to monitor all running searches on a search head and their memory usage in real-time?

benjaminlin1019
Explorer

I use one of the S.O.S. queries to get top 20 memory usage queries every 5 minutes, however, it might be easier for us to see that in job inspection. Does anyone know how you can monitor all the running searches and their memory usage in real-time? I would like to see something similar to 'top' results. thanks

yannK
Splunk Employee
Splunk Employee

SOS scripts is a good start.

Otherwise on recent splunk you can use the introspection logs, they run every 30 seconds.
Pick an host (indexer or search-head) and look at the cpu/mem usage per search pid.

index=_introspection host=* source=*/resource_usage.log* | spath | search component=PerProcess                 
    | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval process_class = case( process=="mongod","KV store", process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input") | eval process_class = case( process=="splunkd" AND ((like(args,"-p %start%") AND NOT like(args,"%process-runner%")) OR args=="service"),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class)
| search process_class="search"                     | stats latest(data.pct_cpu) AS resource_usage_cpu_dedup latest(data.mem_used) AS resource_usage_mem_dedup latest(process_class) AS process_class by data.pid,  _time, data.search_props.type,data.search_props.mode,  data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid                   

benjaminlin1019
Explorer

Thanks, I could not find that particular index exist, do you know that any configuration I need to enable first?

0 Karma

yannK
Splunk Employee
Splunk Employee

My bad, it's an underscore index : index=_introspection

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@yannK and @benjaminlin1019 - Search with some performance enhancement. Please check it out if you wish.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
  | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...