Monitoring Splunk

Can we find out what the memory usage is for our queries, avg and max per day?

danielbb
Motivator

We would like to implement the recommended indexer's memory protection as @somesoni2 recommended at What are the proper user quotas to protect our indexers?

Before we do it, is there a way to find out what the memory usage is for our queries, avg and max per day?

Labels (1)
Tags (2)
0 Karma

aberkow
Builder

If you're an admin in your instance, you should have access to the Splunk Monitoring Console - https://docs.splunk.com/Documentation/Splunk/8.0.1/DMC/DMCoverview. In one of the tabs (at $SPLUNK_HOME/en-US/app/splunk_monitoring_console/search_activity_instance?), you can view various historical views on different query statistics, one of which is "Resource Usage of Searches".

The search under the hood is doing something like this:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid 
| stats sum(resource_usage_dedup) AS sum_resource_usage by _time, type 
| eval sum_resource_usage=round((sum_resource_usage / 100.0),2) 
| timechart minspan=10s Median(sum_resource_usage) AS "Median of resource usage" by type

but you could easily configure this to get the max, avg, etc of different resource counters by configuring the UI or directly altering the search with different resource counters and removing a few lines:

(data.search_props.sid::* component=PerProcess host=SPLUNKSEARCH index=_introspection sourcetype=splunk_resource_usage) 
| eval pid='data.pid', pct_cpu='data.pct_cpu', sid='data.search_props.sid', type='data.search_props.type' 
| bin _time span=10s 
| stats latest(pct_cpu) AS resource_usage_dedup by _time, type, sid, pid
| stats max(resource_usage_dedup), avg(resource_usage_dedup)

Hope this helps!

Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...