Splunk Search

Search performance impact & How to find user deploying high impact searches

mvagionakis
Path Finder

Hello Splunkers, I have two questions today, concerning user's queries and performance impact.

I couldn't find a clear answer in the forum (unless I didn't search very well...).

Is there any way to find if a user launches a search that has a high impact on my indexers' system performance?

Several times in the month I have RAM+CPU saturation on my indexers (not on the same and not in the same frequency).
At this moment I see many splunkd processes running but I'd like to find out if it is a "heavy" query launched by a user and so to inform him.

The goal is to educate users by applying Splunk's best practices.

The last question:
Is there any way to restrict users using the "*" character in their queries or stop using it if it's not in the logic "key=value"?

Thank you in advance.
Michael

0 Karma
1 Solution

VatsalJagani
SplunkTrust
SplunkTrust

Hi @mvagionakis

Answer to your first question, to see resource utilization go to Settings > Monitoring Console > Resource Usage and below search query which shows which user executed search how much CPU and memory it used.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

Answer to your second question is No, you cannot restrict a user from using *. But you can restrict the user to access particular indexes only when they say index=* by setting up default searchable indexes. Settings > Access Controls > Roles > Select User, Indexes searched by default. If you want to restrict some user to completely search the index you can use "Indexes" on the same setting page.

Hope this helps!!!

View solution in original post

gjanders
SplunkTrust
SplunkTrust

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshooting_indexer_cpu
troubleshooting_resource_usage_per_user
detect_excessive_search_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Hi @mvagionakis

Answer to your first question, to see resource utilization go to Settings > Monitoring Console > Resource Usage and below search query which shows which user executed search how much CPU and memory it used.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

Answer to your second question is No, you cannot restrict a user from using *. But you can restrict the user to access particular indexes only when they say index=* by setting up default searchable indexes. Settings > Access Controls > Roles > Select User, Indexes searched by default. If you want to restrict some user to completely search the index you can use "Indexes" on the same setting page.

Hope this helps!!!

mvagionakis
Path Finder

Hey VatsalJagani,

that's almost what I need.
With your query I can find everything except the user's query but I can find it with another query that I found in this forum.

Thank you very much.
Michael

0 Karma

CK_Gold
Loves-to-Learn

Hi,

I have a similar situation as yours. I want to find users who perform searches that are resource intensive. Could you share the search strings you used to perform your task?

Thanks

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Nice to here that. You can comment that query to find search query down here so in future it helps other users like us.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...