Hi
I would like to know the hardware requirements with regards to Splunk Search Head(s) and Indexers to support concurrent users numbering 20,000 to 30,000 users.
Please let me know.
Thanks
Its depend on your server hardware model and type, Its all depend on how much memory you have on server how many users are running searches etc.Lets you have 8 core 24GRam right for 40 users one search head is enough.
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
To fix the problem more generally, you can tweak some configuration nobs in limits.conf as follows:
a) max_searches_per_cpu
While increasing this could fix the dashboard issue where searches are fairly cheap to run, this could lead to performance degradation if you've scheduled a large number of expensive searches.
b) dispatch_quota_retry
This is the number of retries the back end will attempt before throwing the quota/limit error. The back end here does an exponential back-off starting with 100ms and doubling that every time it retries.
c) dispatch_quota_sleep_ms
The initial sleep time for retries. Instead of increasing max_searches_per_cpu you can set the dispatch_quota_retry to 10 which will instruct the back-end to retry dispatching a particular search for about 100 seconds before throwing the quota/limit error.