Splunk Search

Does anyone know how to monitor all running searches on a search head and their memory usage in real-time?

benjaminlin1019
Explorer

I use one of the S.O.S. queries to get top 20 memory usage queries every 5 minutes, however, it might be easier for us to see that in job inspection. Does anyone know how you can monitor all the running searches and their memory usage in real-time? I would like to see something similar to 'top' results. thanks

yannK
Splunk Employee
Splunk Employee

SOS scripts is a good start.

Otherwise on recent splunk you can use the introspection logs, they run every 30 seconds.
Pick an host (indexer or search-head) and look at the cpu/mem usage per search pid.

index=_introspection host=* source=*/resource_usage.log* | spath | search component=PerProcess                 
    | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval process_class = case( process=="mongod","KV store", process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input") | eval process_class = case( process=="splunkd" AND ((like(args,"-p %start%") AND NOT like(args,"%process-runner%")) OR args=="service"),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class)
| search process_class="search"                     | stats latest(data.pct_cpu) AS resource_usage_cpu_dedup latest(data.mem_used) AS resource_usage_mem_dedup latest(process_class) AS process_class by data.pid,  _time, data.search_props.type,data.search_props.mode,  data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid                   

benjaminlin1019
Explorer

Thanks, I could not find that particular index exist, do you know that any configuration I need to enable first?

0 Karma

yannK
Splunk Employee
Splunk Employee

My bad, it's an underscore index : index=_introspection

0 Karma

VatsalJagani
Champion

@yannK and @benjaminlin1019 - Search with some performance enhancement. Please check it out if you wish.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
  | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid
0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...