Getting Data In

What are the proper user quotas to protect our indexers?

danielbb
Motivator

Yesterday, one indexer got crashed due to a very badly developed dashboard - it instantly consumed all the memory of the indexer.

Which quotas should we place in order to prevent such cases?

Btw, it does seem that this particular indexer ran all or most of the queries of this dashboard, which is weird.

alt text

Tags (2)
0 Karma

bandit
Motivator

A few things you can look at:
You can modify times.conf and verify users are not searching All time by default
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Timesconf
https://answers.splunk.com/answers/79547/disable-all-time.html

Force users to specify indexes in their queries by verifying that "Indexes searched by default" for the role assigned to the users is NOT set to "All non-internal indexes"

Verify that data is being distributed equally to indexers (more or less) otherwise the workload won't be evenly distributed and you'll be waiting longer for the query to complete if one indexer is doing most of the work. splunk_server is the field name representing the indexer that is hosting/serving the data.

your base query | top 100 splunk_server

somesoni2
SplunkTrust
SplunkTrust

danielbb
Motivator

That's great. Btw, what's the default max memory allocation per a search query?

0 Karma

danielbb
Motivator

@somesoni2 - I see these values on the indexers -

$SPLUNK_HOME/etc/system/default/limits.conf - enable_memory_tracker = false
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_percentage_threshold = 25
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_threshold = 4000

Should we change these values on the indexers and on the SHs as well?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...