topic Re: What are the proper user quotas to protect our indexers? in Getting Data In

What are the proper user quotas to protect our indexers?

danielbb — Wed, 25 Sep 2019 15:27:21 GMT

Yesterday, one indexer got crashed due to a very badly developed dashboard - it instantly consumed all the memory of the indexer.

Which quotas should we place in order to prevent such cases?

Btw, it does seem that this particular indexer ran all or most of the queries of this dashboard, which is weird.

Re: What are the proper user quotas to protect our indexers?

somesoni2 — Wed, 25 Sep 2019 18:40:23 GMT

Have a look at this:
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Limitsearchprocessmemoryusage

Re: What are the proper user quotas to protect our indexers?

danielbb — Wed, 25 Sep 2019 20:58:54 GMT

That's great. Btw, what's the default max memory allocation per a search query?

Re: What are the proper user quotas to protect our indexers?

bandit — Thu, 26 Sep 2019 03:55:01 GMT

A few things you can look at:
You can modify times.conf and verify users are not searching All time by default
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Timesconf
https://answers.splunk.com/answers/79547/disable-all-time.html

Force users to specify indexes in their queries by verifying that "Indexes searched by default" for the role assigned to the users is NOT set to "All non-internal indexes"

Verify that data is being distributed equally to indexers (more or less) otherwise the workload won't be evenly distributed and you'll be waiting longer for the query to complete if one indexer is doing most of the work. splunk_server is the field name representing the indexer that is hosting/serving the data.

your base query | top 100 splunk_server

Re: What are the proper user quotas to protect our indexers?

danielbb — Fri, 27 Dec 2019 16:12:20 GMT

@somesoni2 - I see these values on the indexers -

$SPLUNK_HOME/etc/system/default/limits.conf - enable_memory_tracker = false
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_percentage_threshold = 25
$SPLUNK_HOME/etc/system/default/limits.conf - search_process_memory_usage_threshold = 4000

Should we change these values on the indexers and on the SHs as well?