If you're an admin in your instance, you should have access to the Splunk Monitoring Console - https://docs.splunk.com/Documentation/Splunk/8.0.1/DMC/DMCoverview. In one of the tabs (at $SPLUNK_HOME/en-US/app/splunk_monitoring_console/search_activity_instance?), you can view various historical views on different query statistics, one of which is "Resource Usage of Searches".
The search under the hood is doing something like this:
(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage)
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type'
| bin _time span=10s
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid
| stats sum(resource_usage_dedup) AS sum_resource_usage by _time, type
| eval sum_resource_usage=round((sum_resource_usage / 100.0),2)
| timechart minspan=10s Median(sum_resource_usage) AS "Median of resource usage" by type
but you could easily configure this to get the max, avg, etc of different resource counters by configuring the UI or directly altering the search with different resource counters and removing a few lines:
(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage)
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type'
| bin _time span=10s
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid
| stats max(resource_usage_dedup), avg(resource_usage_dedup)
Hope this helps!
... View more