How best to use two servers to scale Splunk from a single instance?

[bdruth]

We're looking at an initial scale-out of Splunk, beyond the traditional, initial single Splunk Enterprise search head + indexer in one. We've essentially been given two servers to move our current server to. I can think of a few permutations, but I'm wondering what's a recommended configuration:

1. server 1: indexer, server 2: search head
2. server 1: indexer + search head, server 2: search head
3. server 1: indexer + search head, server 2: indexer + search head
4. server 1: indexer, server 2: indexer + search head

All our forwarders are under provisioning automation, so changing the configuration to load balance to two servers shouldn't be a problem, and either way, we have to push out a configuration change for a new indexer. I'm not sure how load balancing to the search heads works - but we have the ability to setup a LB pool on the F5, if that makes sense.

Our indexing license is 100GB/day, and the two server nodes are VMs in a larger vSphere/vBlock deployment. 4 CPUs x 8GB RAM, each. Each has ample disk space to be an indexer. We run a fair number of scheduled searches and when we're investigating an issue / outage, we run a fair number of real-time searches, which tend to start bogging down the current single server we have. I've noticed high load on that server when we run a one-shot on the forwarding servers that pushes a lot of data at one time to the indexer as well.

So, that's all to say, it's not clear to us if we need more indexing capacity, more search head capacity, or just more of both. I'm kind of leaning to the latter. For this reason, we're thinking option #3 (above) might be the place to start, until we have a clearer indicating one way or the other.

I can't find clear docs indicating if #3 is a valid configuration or what to put in place to make it work. Ultimately, we want to be able to still search across the entire index and share the search load across both servers.

Thanks!

p.s. I thought I posted this a couple days ago, but it's not showing up in my profile or in searching answers.splunk.com. Sorry for the repost if it did in fact post mysteriously.

[masonmorales]

The general rule of thumb is to add indexers before you add search heads, because additional indexers allow searches to complete faster via distributed search. Faster searches mean that you have less concurrent searches on your search head, reducing your need for additional search heads.

I'd recommend option 4:

server 1: indexer, server 2: indexer + search head

As someone else already mentioned, you should also consider adding cores and RAM to improve performance. See: http://docs.splunk.com/Documentation/Splunk/6.2.2/Capacity/Referencehardware

[s2_splunk · Accepted solution]

Just because you have two servers doesn't necessarily mean that you can improve your deployment by using them both. If you are currently not reaching any scalability limits on ingest and search, a single server will give you the best possible performance, since you are not introducing distributed search.

You are under-provisioned from a core perspective. I am assuming that your 4 vCPUs are also not reserved (same for RAM), and your indexers compete for CPU resources with other VMs.
I think your best bet at given volumes is to get a VM with at least 8 vCPUs and at least 12GB of RAM and convince your VM admins to reserve those resources (which will be a battle).

All your configs are valid; you can have to SH/IDXr combos, where the indexers are search peers to each other. You don't improve availability with it and you are certainly not increasing capacity, because searches will now compete for resources with indexers.

If you cannot follow our guidelines here I would at the very least recommend getting a third VM (with 8 or more cores) to be the SH in front of your indexers. But again, your total core count is not really sufficient to provide enough compute on the indexers to do both, indexing & searching (every search started on the SH will consume an indexer core).

this may also be helpful.

[Raghav2384]

Hi, The combination would depend on your requirements. 

Search Heads: CPU Related
Indexers: Need good IOPS as here's where the magic of splunk happens

Now, what's the user base? if you have too many users, then using two search heads (partial) is a good idea. Considering concurrent, rt searches and the whole 9 yards.

I see license is 100GB / day...what's the IOPS ? if it has low IOPS, spreading the load across two Indexers is ideal. Now, what type of data you are going to index? Is it well structured,key-valued data or needle in the haystack?

I would suggest you to consider the requirements and that will definitely help you determine the topology
 1. Use cases, how many users, concurrent users.
 2. Amount of data to be indexed. Type of logs, Specs etc.

Hope this helps!
Thanks,
Raghav

[bdruth]

Thank you!