Can we find out what the memory usage is for our queries, avg and max per day?

[danielbb]

We would like to implement the recommended indexer's memory protection as @somesoni2 recommended at What are the proper user quotas to protect our indexers?

Before we do it, is there a way to find out what the memory usage is for our queries, avg and max per day?

[aberkow]

If you're an admin in your instance, you should have access to the Splunk Monitoring Console - https://docs.splunk.com/Documentation/Splunk/8.0.1/DMC/DMCoverview. In one of the tabs (at $SPLUNK_HOME/en-US/app/splunk_monitoring_console/search_activity_instance?), you can view various historical views on different query statistics, one of which is "Resource Usage of Searches".

The search under the hood is doing something like this:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid 
| stats sum(resource_usage_dedup) AS sum_resource_usage by _time, type 
| eval sum_resource_usage=round((sum_resource_usage / 100.0),2) 
| timechart minspan=10s Median(sum_resource_usage) AS "Median of resource usage" by type

but you could easily configure this to get the max, avg, etc of different resource counters by configuring the UI or directly altering the search with different resource counters and removing a few lines:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid
| stats max(resource_usage_dedup), avg(resource_usage_dedup)

Hope this helps!