Splunk Search

Recommended No of Concurrent searches for the current infrastructure



We have a single server machine, where Splunk enterprise edition is installed.
CPU - 1
Cores - 8
RAM - 32 GB

We have implemented several dashboards, charts and tables. Also, where more than 5 users will concurrently access these datas. Due to which, we faced lot of performance issue, such as waiting for queue or maximum concurrent search was reached.

Because of which we were forced to change the default configuration in limit.conf and update the following attributes,

Now, according to the formula,
No of Concurrent searches = 4 *(1*8) + 6 = 38 (as per my understanding it will handle 38 searches concurrently)

After the changes,
Where it reaches max of 56 concurrent searches and CPU usage of around 80 to 90 percent.

Is this configuration is recommended? please suggest the nos for the attributes or alternate approach for the current system configurations?

Many thanks.

0 Karma


"Recommended?" I don't think you'll get anyone to say it's actually recommended.

"Good enough?" I'd say so, with a few big caveats:

1) You are running that server pretty hard. There's essentially zero headroom for a spike in data ingestion, an extra user hopping on, or pretty much anything else. Which is fine, I guess, as long as you are aware that this is happening and that the moment this server "gets behind" for an hour, it may never catch up again because it has no spare capacity. 😞

2) max_searches_per_cpu=6 is pretty high, in my opinion. But not so high it would make support cry. At least I don't think.

3) Sort of a follow on to the above, maybe you just need more server? Or to split it into an indexer and separate SH - then you can concentrate IO on the IDX and use a cheaper less IO-capable SH.

3.1) Or, you might also get some significant advantages from fine-tuning searches. Optimizing those might make this whole situation moot by making the resource utilization drop by half or more.

But in any case, what it comes down to is that it's really what works for you, and this seems to work for you. While maybe stretching things a bit, it isn't so far out on the fringe that it's just "obviously wrong".

Note - all the above is my opinion, and my opinion only. If these settings make your server catch fire and explode, makes your puppy sick or otherwise creates a discontinuity in the space-time continuum, well, it's not my fault.

Extra note - on the other hand, if you love it, and are amazed, and it works well for you for a long time, then I'll happily accept credit for it. Or at least click the "Accept" button on the answer. 🙂

Happy Splunking!

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...