Monitoring Splunk
Highlighted

Can we find out what the memory usage is for our queries, avg and max per day?

Motivator

We would like to implement the recommended indexer's memory protection as @somesoni2 recommended at What are the proper user quotas to protect our indexers?

Before we do it, is there a way to find out what the memory usage is for our queries, avg and max per day?

Labels (1)
Tags (2)
0 Karma
Highlighted

Re: Can we find out what the memory usage is for our queries, avg and max per day?

Builder

If you're an admin in your instance, you should have access to the Splunk Monitoring Console - https://docs.splunk.com/Documentation/Splunk/8.0.1/DMC/DMCoverview. In one of the tabs (at $SPLUNKHOME/en-US/app/splunkmonitoringconsole/searchactivity_instance?), you can view various historical views on different query statistics, one of which is "Resource Usage of Searches".

The search under the hood is doing something like this:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid 
| stats sum(resource_usage_dedup) AS sum_resource_usage by _time, type 
| eval sum_resource_usage=round((sum_resource_usage / 100.0),2) 
| timechart minspan=10s Median(sum_resource_usage) AS "Median of resource usage" by type

but you could easily configure this to get the max, avg, etc of different resource counters by configuring the UI or directly altering the search with different resource counters and removing a few lines:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid
| stats max(resource_usage_dedup), avg(resource_usage_dedup)

Hope this helps!

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.