Splunk Search

Recommended No of Concurrent searches for the current infrastructure



We have a single server machine, where Splunk enterprise edition is installed.
CPU - 1
Cores - 8
RAM - 32 GB

We have implemented several dashboards, charts and tables. Also, where more than 5 users will concurrently access these datas. Due to which, we faced lot of performance issue, such as waiting for queue or maximum concurrent search was reached.

Because of which we were forced to change the default configuration in limit.conf and update the following attributes,

Now, according to the formula,
No of Concurrent searches = 4 *(1*8) + 6 = 38 (as per my understanding it will handle 38 searches concurrently)

After the changes,
Where it reaches max of 56 concurrent searches and CPU usage of around 80 to 90 percent.

Is this configuration is recommended? please suggest the nos for the attributes or alternate approach for the current system configurations?

Many thanks.

0 Karma


"Recommended?" I don't think you'll get anyone to say it's actually recommended.

"Good enough?" I'd say so, with a few big caveats:

1) You are running that server pretty hard. There's essentially zero headroom for a spike in data ingestion, an extra user hopping on, or pretty much anything else. Which is fine, I guess, as long as you are aware that this is happening and that the moment this server "gets behind" for an hour, it may never catch up again because it has no spare capacity. 😞

2) max_searches_per_cpu=6 is pretty high, in my opinion. But not so high it would make support cry. At least I don't think.

3) Sort of a follow on to the above, maybe you just need more server? Or to split it into an indexer and separate SH - then you can concentrate IO on the IDX and use a cheaper less IO-capable SH.

3.1) Or, you might also get some significant advantages from fine-tuning searches. Optimizing those might make this whole situation moot by making the resource utilization drop by half or more.

But in any case, what it comes down to is that it's really what works for you, and this seems to work for you. While maybe stretching things a bit, it isn't so far out on the fringe that it's just "obviously wrong".

Note - all the above is my opinion, and my opinion only. If these settings make your server catch fire and explode, makes your puppy sick or otherwise creates a discontinuity in the space-time continuum, well, it's not my fault.

Extra note - on the other hand, if you love it, and are amazed, and it works well for you for a long time, then I'll happily accept credit for it. Or at least click the "Accept" button on the answer. 🙂

Happy Splunking!

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...