Getting Data In

What are the proper user quotas to protect our indexers?

danielbb
Motivator

Yesterday, one indexer got crashed due to a very badly developed dashboard - it instantly consumed all the memory of the indexer.

Which quotas should we place in order to prevent such cases?

Btw, it does seem that this particular indexer ran all or most of the queries of this dashboard, which is weird.

alt text

Tags (2)
0 Karma

bandit
Motivator

A few things you can look at:
You can modify times.conf and verify users are not searching All time by default
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Timesconf
https://answers.splunk.com/answers/79547/disable-all-time.html

Force users to specify indexes in their queries by verifying that "Indexes searched by default" for the role assigned to the users is NOT set to "All non-internal indexes"

Verify that data is being distributed equally to indexers (more or less) otherwise the workload won't be evenly distributed and you'll be waiting longer for the query to complete if one indexer is doing most of the work. splunk_server is the field name representing the indexer that is hosting/serving the data.

your base query | top 100 splunk_server

somesoni2
Revered Legend

danielbb
Motivator

That's great. Btw, what's the default max memory allocation per a search query?

0 Karma

danielbb
Motivator

@somesoni2 - I see these values on the indexers -

$SPLUNK_HOME/etc/system/default/limits.conf - enable_memory_tracker = false
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_percentage_threshold = 25
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_threshold = 4000

Should we change these values on the indexers and on the SHs as well?

0 Karma
Get Updates on the Splunk Community!

Unlock New Opportunities with Splunk Education: Explore Our Latest Courses!

At Splunk Education, we’re dedicated to providing top-tier learning experiences that cater to every skill ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...