topic Re: Search performance impact & How to find user deploying high impact searches in Splunk Search

Search performance impact & How to find user deploying high impact searches

mvagionakis — Wed, 05 Jun 2019 10:58:42 GMT

Hello Splunkers, I have two questions today, concerning user's queries and performance impact.

I couldn't find a clear answer in the forum (unless I didn't search very well...).

Is there any way to find if a user launches a search that has a high impact on my indexers' system performance?

Several times in the month I have RAM+CPU saturation on my indexers (not on the same and not in the same frequency).
At this moment I see many splunkd processes running but I'd like to find out if it is a "heavy" query launched by a user and so to inform him.

The goal is to educate users by applying Splunk's best practices.

The last question:
Is there any way to restrict users using the "*" character in their queries or stop using it if it's not in the logic "key=value"?

Thank you in advance.
Michael

Re: Search performance impact & How to find user deploying high impact searches

VatsalJagani — Wed, 05 Jun 2019 13:33:15 GMT

Hi @mvagionakis

Answer to your first question, to see resource utilization go to Settings > Monitoring Console > Resource Usage and below search query which shows which user executed search how much CPU and memory it used.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

Answer to your second question is No, you cannot restrict a user from using *. But you can restrict the user to access particular indexes only when they say index=* by setting up default searchable indexes. Settings > Access Controls > Roles > Select User, Indexes searched by default. If you want to restrict some user to completely search the index you can use "Indexes" on the same setting page.

Hope this helps!!!

Re: Search performance impact & How to find user deploying high impact searches

mvagionakis — Wed, 05 Jun 2019 14:55:56 GMT

Hey VatsalJagani,

that's almost what I need.
With your query I can find everything except the user's query but I can find it with another query that I found in this forum.

Thank you very much.
Michael

Re: Search performance impact & How to find user deploying high impact searches

VatsalJagani — Wed, 05 Jun 2019 15:01:27 GMT

Nice to here that. You can comment that query to find search query down here so in future it helps other users like us.

Re: Search performance impact & How to find user deploying high impact searches

gjanders — Wed, 30 Sep 2020 00:49:47 GMT

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshooting_indexer_cpu
troubleshooting_resource_usage_per_user
detect_excessive_search_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

Re: Search performance impact & How to find user deploying high impact searches

CK_Gold — Fri, 15 Sep 2023 13:42:14 GMT

Hi,

I have a similar situation as yours. I want to find users who perform searches that are resource intensive. Could you share the search strings you used to perform your task?

Thanks