Splunk Search

How to track memory/cpu usage per search execution (on Search Head/Indexer)?

melonman
Motivator

Hi

I am looking for a way to track memory/cpu usage per search execution on search head and indexer.
I thought I could use _introspection index to track it, but I can not find process resource information in there.

I am currenly testing with splunk6.4.0 on MacOS, and I am trying to monitor searches on Search head Splunk 6.3.2 and Indexer Splunk 6.0.2.

Could anyone comment on this?

Thank you,

gjanders
SplunkTrust
SplunkTrust

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshooting_indexer_cpu
troubleshooting_resource_usage_per_user
detect_excessive_search_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

spunk311z
Path Finder

Lots of great info and search queries in this thread (thanks),  splunk really is amazing!

One thing i can contribute is this search (below) that i often use to show all of my scheduled reports (it pairs nicely with some of the resource usage searches in this thread to help ID and modify your scheduled reports or their cron entry).

Also its nice to review this from time to time as its easy to loose track of cron scheduled reports you may no longer need to run (or run as frequently);

| rest /servicesNS/-/-/saved/searches  | search is_scheduled=1 | table author cron_schedule is_scheduled schedule_window title updated embed.enabled Search

 thanks

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Hi @melonman,

If you want to search CPU and memory utilization per search execution with relevant information like which used executed and more.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
 | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

MuS
SplunkTrust
SplunkTrust

Hi melonman,

Did you check out the Distributed Management Console http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/DMCoverview this should provide data for the search head.
Regarding the indexer try this search

host=YourHostNameHere sourcetype=splunk_resource_usage index=_introspection component=PerProcess "data.process_type"=search

Hope this helps ...

cheers, MuS

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...