topic How do you identify which saved searches are consuming max resources to monitor on a dashboard panel? in Dashboards & Visualizations

How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

kamal_jagga — Tue, 09 Jun 2015 08:12:08 GMT

I have 65 Saved searches.

Now, I want to build a dashboard panel in which top 10 searches consuming max resources can be depicted. Kindly advise.

I have read answers to other questions also, but no where any specific search is mentioned.

Kindly advise.

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

David — Tue, 09 Jun 2015 15:46:35 GMT

Two options for you. The specific answer to your question is:

index=_internal savedsearch_name=* NOT savedsearch_name="" sourcetype=splunk_audit OR sourcetype=audittrail | stats count sum(total_run_time) avg(total_run_time) avg(scan_count) avg(event_count) by savedsearch_name | sort 10 - "sum(total_run_time)"

That said, if you want to do large scale analysis on your search logs, I recommend checking out my app, Search Activity. The current version of the app doesn't have a report (I'm adding it into the next version). However, you can run the report with the following search:

| tstats count sum(total_run_time) avg(total_run_time) avg(scan_count) avg(event_count) values(user) from `SA_SearchHistory` where searchtype=scheduled groupby savedsearch_name | sort 10 - "sum(total_run_time)"

The benefit of my app for this analysis (in addition to all the other visibility you can get) is that in my lab, it is over 50 times faster. If you want to do more, or the first search isn't fast enough, check out the app.

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

kamal_jagga — Mon, 28 Sep 2020 20:12:05 GMT

Hi,

The above search provided by you didn't give the results. But i removed some portion of the string and got results with the below mentioned query.

index=_internal savedsearch_name=* NOT savedsearch_name="" | stats count by savedsearch_name | sort 30

But it still didn't give the metrics of the resource consumption by individual searches.

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

epeeran — Tue, 29 Sep 2020 22:30:46 GMT

index=_internal savedsearch_name=* NOT savedsearch_name="" | stats count by savedsearch_name | sort count | head 30

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

epeeran — Tue, 29 Sep 2020 22:30:48 GMT

index=_internal savedsearch_name=* NOT savedsearch_name="" | stats count by savedsearch_name | sort count | head 30

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

epeeran — Tue, 29 Sep 2020 22:30:51 GMT

index=_internal savedsearch_name=* NOT savedsearch_name="" | stats count by savedsearch_name | sort count | head 30

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

epeeran — Tue, 29 Sep 2020 22:30:54 GMT

index=_internal savedsearch_name=* NOT savedsearch_name="" | stats count by savedsearch_name | sort count | head 30

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

prakash007 — Tue, 29 Sep 2020 22:30:56 GMT

@kamal_jagga : If you have a DMC configured in your environment, there are prebuilt dashboards in there...
DMC-->search--->activity--->search activity:instance--->select your SearchHead instance(Top 20 Memory-Consuming Searches if you scroll down). you can use that underlying search with some tweaks to build a dashboard.
https://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/SearchactivityDeploymentwide#Interpret_results_in_these_dashboards

Re: How do you identify which saved searches are consuming max resources to monitor on a dashboard panel?

gjanders — Tue, 18 Dec 2018 22:17:32 GMT

In the app Alerts for Splunk Admins or github links to the dashboards I've got a few dashboards that could help here.

Troubleshooting indexer CPU & Troubleshooting resource usage per user, you could tweak either of those to look for the search id's with the scheduler in the name...they measure CPU usage , memory usage et cetera.

As pointed out in another post you can get some of this information via the monitoring console as well, it depends on how much information you want about the impact of scheduled searches (or searches in general)