Splunk Search
Highlighted

Search performance impact & How to find user deploying high impact searches

Path Finder

Hello Splunkers, I have two questions today, concerning user's queries and performance impact.

I couldn't find a clear answer in the forum (unless I didn't search very well...).

Is there any way to find if a user launches a search that has a high impact on my indexers' system performance?

Several times in the month I have RAM+CPU saturation on my indexers (not on the same and not in the same frequency).
At this moment I see many splunkd processes running but I'd like to find out if it is a "heavy" query launched by a user and so to inform him.

The goal is to educate users by applying Splunk's best practices.

The last question:
Is there any way to restrict users using the "*" character in their queries or stop using it if it's not in the logic "key=value"?

Thank you in advance.
Michael

0 Karma
Highlighted

Re: Search performance impact & How to find user deploying high impact searches

Motivator

Hi @mvagionakis

Answer to your first question, to see resource utilization go to Settings > Monitoring Console > Resource Usage and below search query which shows which user executed search how much CPU and memory it used.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

Answer to your second question is No, you cannot restrict a user from using *. But you can restrict the user to access particular indexes only when they say index=* by setting up default searchable indexes. Settings > Access Controls > Roles > Select User, Indexes searched by default. If you want to restrict some user to completely search the index you can use "Indexes" on the same setting page.

Hope this helps!!!

View solution in original post

Highlighted

Re: Search performance impact & How to find user deploying high impact searches

Path Finder

Hey VatsalJagani,

that's almost what I need.
With your query I can find everything except the user's query but I can find it with another query that I found in this forum.

Thank you very much.
Michael

0 Karma
Highlighted

Re: Search performance impact & How to find user deploying high impact searches

Motivator

Nice to here that. You can comment that query to find search query down here so in future it helps other users like us.

0 Karma
Highlighted

Re: Search performance impact & How to find user deploying high impact searches

SplunkTrust
SplunkTrust

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshootingindexercpu
troubleshootingresourceusageperuser
detectexcessivesearch_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

0 Karma