Quick Question - CPU Search Limit on Search Head or Indexer?

[aferone]

We are running into max concurrent searches issues, as our deployment is getting more and more used. Is the limit based on the Search Head CPUs, or the Indexer CPUs?

Is it worth building more Search Heads?

Thanks!

[aferone]

We do have plans to add another indexer. Good to know we will get benefits from that. And I will try the increase and monitor the indexer.

Thanks for all of the help to everyone!

[martin_mueller]

The font increased because the pound sign marks a comment in the config but a heading in here 🙂

As it is now, you have 16 cores on the SH side and 12 cores + 1xIO on the IDX side. Usually I'd expect IDX IO to be the bottleneck, adding more cores to the SH would not fix that. Adding a second IDX would make searches complete faster, hence reducing concurrency and improving user experience.

Your situation may be non-standard of course, so do try increasing the per-CPU searches from 1 to 2. Monitor your Indexer to see if you're pounding it with too many searches or not.

[gkanapathy]

Again, the settings here are just fiddling with software limits, it doesn't actually help your searches run faster and it won't help if you're already at max utilization on the indexer.

[gkanapathy]

If you only have a single indexer, an additional search head won't help. Adding an additional indexer probably will help by reducing the concurrency by speeding up search times (for most searches...this does depend on your searches).

In most cases you don't get better performance with 1 SH + 1 indexer vs a single instance combined SH+IDX. Often, performance will be worse. The best next step after a single-node instance is actually 1 indexer + 1 combined. It's probably not till you get to having three separate indexers does it become worthwhile to add a standalone search head.

[aferone]

Not sure why the font increased on that last reply. My apologies!

Yes, the search head has 16 cores.

So you're saying that adding a totally separate Search Head won't work in this case?

[martin_mueller]

Doing the maths on that means your SH would have 16 cores. Make sure that's accurate.

If that's the case then you can set max_searches_per_cpu = 2 safely from the SH point of view, but I suspect that's then going to swamp the single indexer so in case of overutilization there you'll greatly benefit from a second indexer.

[aferone]

Our indexer and Search Head both have the same settings:

the maximum number of concurrent searches per CPU

max_searches_per_cpu = 1

the base number of concurrent searches

base_max_searches = 6

[martin_mueller]

Ah, so you're already higher than the current default values. I guess you have base_max_searches = 6 and max_searches_per_cpu = 4?

How's the utilization per core on your SH and Indexer as well as IO on the Indexer while you're near that limit? If those three all have room to go then you could go higher, else add HW where there's not a lot of room.

[aferone]

The error says maximum is 22.

"The system is approaching the maximum number of historical searches that can be run concurrently. current=20 maximum=22"

[martin_mueller]

Using the default settings, your four-core SH would allow 10 searches (6 + 1*4). You're probably safe to increase the max_searches_per_cpu from 1 to 2 in limits.conf, giving you 14. Monitor load on the indexer, especially check if indexing gets backed up (SoS app).

Adding another SH and moving scheduled searches for dashboards to that won't work without a SH pool, the physical SH won't be able to use the results. However, you can move alerts and summary searches to a second SH without the need for a pool - just make sure configs such as field extractions are in sync.

[gkanapathy]

It is based on the CPU of the search head. Note that the limit is a software limit, and may not correspond to your hardware capacity, i.e., depending on the searches you are running, your hardware may be able to run more; or you may have passed the point of efficiency a long time ago and have simply been waiting a long time for searches to run (Just because the system lets you run more concurrent jobs doesn't mean it completes more jobs in the same amount of time, as each job may simply take longer to complete.)

Note also that increasing the number of searches allowed on the search head, even if you have more capacity on the search head, may cause over-utilization of the indexers and you may run out of capacity there, and still not be able to run any more searches in the same amount of time.

[aferone]

We're hitting the "The system is approaching the maximum number of historical searches that can be run concurrently" message. Hardware performance looks fine.

I can immediately add another virtual search head. We can move all of our scheduled searches for our dashboards to that, and let our users use the existing search head.

Does that make sense?

[martin_mueller]

Are you reaching the configured limit of concurrent searches or the performance limit of your hardware?

As for adding HW, based on your figures you have three options:

• add another indexer - helps with everything except limits.conf
• add cores to your SH - helps with concurrent search limits and search performance, may not be easy to do with physical hardware
• add another SH - helps with concurrent search limits and search performance, but requires SH pooling and a load balancer or splitting SHs into a user SH and a job SH.

There's no definitive "gap" in your HW, so each may work.

[aelliott]

As martin said, it depends on the search head core count, so adding more cores to the search head will give you more searches, I would add more cores to your existing search head if possible.- see http://docs.splunk.com/Documentation/Splunk/6.1.1/DistSearch/Configuredistributedsearch

[aferone]

I guess that still doesn't clear up my confusion. If I created 2 more virtual Search Heads, would that help? Or do I need to add an indexer? Right now, we have 1 indexer with 2 physical processors, 6 cores apiece (12 cores). The search head has 1 physical processor and 4 cores.

[martin_mueller]

The value in limits.conf depends on the Search Head core count. Increasing that will allow more searches to run concurrently, but may not speed up overall user experience past some point.

Where to add hardware with the best cost-effectiveness depends on what you have now.

[somesoni2]

I believe it depends on the Indexer CPU. Check this link

http://docs.splunk.com/Documentation/Splunk/6.1.2/Deploy/Accommodatemanysimultaneoussearches

[aferone]

I'll have to confirm that, but does it depend on both? Or just one of them?

[strive]

What is the current CPU and RAM configurations on your SH and indexers?