Search performance impact & How to find user deploying high impact searches

[mvagionakis]

Hello Splunkers, I have two questions today, concerning user's queries and performance impact.

I couldn't find a clear answer in the forum (unless I didn't search very well...).

Is there any way to find if a user launches a search that has a high impact on my indexers' system performance?

Several times in the month I have RAM+CPU saturation on my indexers (not on the same and not in the same frequency).
At this moment I see many splunkd processes running but I'd like to find out if it is a "heavy" query launched by a user and so to inform him.

The goal is to educate users by applying Splunk's best practices.

The last question:
Is there any way to restrict users using the "*" character in their queries or stop using it if it's not in the logic "key=value"?

Thank you in advance.
Michael

[gjanders]

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshooting_indexer_cpu
troubleshooting_resource_usage_per_user
detect_excessive_search_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

[VatsalJagani · Accepted solution]

Hi @mvagionakis

Answer to your first question, to see resource utilization go to Settings > Monitoring Console > Resource Usage and below search query which shows which user executed search how much CPU and memory it used.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

Answer to your second question is No, you cannot restrict a user from using *. But you can restrict the user to access particular indexes only when they say index=* by setting up default searchable indexes. Settings > Access Controls > Roles > Select User, Indexes searched by default. If you want to restrict some user to completely search the index you can use "Indexes" on the same setting page.

Hope this helps!!!

[mvagionakis]

Hey VatsalJagani,

that's almost what I need.
With your query I can find everything except the user's query but I can find it with another query that I found in this forum.

Thank you very much.
Michael

[CK_Gold]

Hi,

I have a similar situation as yours. I want to find users who perform searches that are resource intensive. Could you share the search strings you used to perform your task?

Thanks

[VatsalJagani]

Nice to here that. You can comment that query to find search query down here so in future it helps other users like us.