Splunk Search

Does anyone know how to monitor all running searches on a search head and their memory usage in real-time?

benjaminlin1019
Explorer

I use one of the S.O.S. queries to get top 20 memory usage queries every 5 minutes, however, it might be easier for us to see that in job inspection. Does anyone know how you can monitor all the running searches and their memory usage in real-time? I would like to see something similar to 'top' results. thanks

yannK
Splunk Employee
Splunk Employee

SOS scripts is a good start.

Otherwise on recent splunk you can use the introspection logs, they run every 30 seconds.
Pick an host (indexer or search-head) and look at the cpu/mem usage per search pid.

index=_introspection host=* source=*/resource_usage.log* | spath | search component=PerProcess                 
    | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval process_class = case( process=="mongod","KV store", process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input") | eval process_class = case( process=="splunkd" AND ((like(args,"-p %start%") AND NOT like(args,"%process-runner%")) OR args=="service"),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class)
| search process_class="search"                     | stats latest(data.pct_cpu) AS resource_usage_cpu_dedup latest(data.mem_used) AS resource_usage_mem_dedup latest(process_class) AS process_class by data.pid,  _time, data.search_props.type,data.search_props.mode,  data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid                   

benjaminlin1019
Explorer

Thanks, I could not find that particular index exist, do you know that any configuration I need to enable first?

0 Karma

yannK
Splunk Employee
Splunk Employee

My bad, it's an underscore index : index=_introspection

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@yannK and @benjaminlin1019 - Search with some performance enhancement. Please check it out if you wish.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
  | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...