Combine search head and indexer in a cluster?

[mshilston]

Hi,

I'm hoping someone can help me to understand if the following is possible with Splunk Enterprise as I'm just learning about the various components for a design I'm writing.

The environment I'm working with will have around 100 Windows VMs initially, growing to 200+, probably across a few domains. There will also be networking devices to support that infrastructure which we want to be able to collect logs from as well.

We are going to have two sites (active / passive) and I have two virtual servers in each site assigned to Splunk, although the DR site will be a standby. I'm trying to work with these two VMs without requesting more, although it may be possible to add more.

From what I've read, I think I would be looking at a Data Consolidation topology i.e. multiple universal forwarders pushing data to an indexer / search server. If I wanted to consider high availability options within a single site, I might want to cluster the indexers, but as I only have two servers I'm wondering if it's possible to create a two node cluster hosting both the search and indexing roles?

I don't think this is possible, as under the 'Cluster Nodes' guidance, it states that "Master nodes, peer nodes, and search heads are all specialized Splunk Enterprise instances. All nodes must reside on separate instances and separate machines." - this looks to me like I wouldn't be able to have a cluster with just two servers regardless.

The other option I'm looking at would be to have one server as an Indexer and one as a search / indexer. Forwarders could then get data into the Indexers from the 100+ VMs.

We do have DR options available and will be backing up, so high availability is not necessarily required and could be added later with increased demand.

I suppose the questions are;
1) Is a clustering solution possible with two servers, or what would the minimum number of servers be? Could it be done with 3; two indexers and one search head, or would I need to have a seperate Master Node as well?
2) Am I right in thinking that if I had one server as an Indexer and one as an Indexer / search server, if the dedicated Indexer went down, the results would be incomplete? Furthermore, if the search server went down search would therefore be totally unavaiiable?
3) Is it possible to have two servers running as both Indexers and Search servers, or would I need a search head to manage both indexers in this case?

Many thanks!

M

[mshilston]

Thanks garethatiag,

Following further reading and having considered again the requirements for what I am trying to achieve, I think I would want to have a resilient search head cluster after all.

I'm looking at this topology now; http://docs.splunk.com/File:SH_cluster_single_site_indexer.png

In this case, I think I am looking at the following;
- 1 x Deployer
- 3 x Search Heads
- 2 x Indexers
- 200+ Universal Forwarders

So... 6 servers?

Only thing I'm not clear on is if I need a separate 'Master Node' for the Indexers as shown, or can one of my two 'Indexers' act as the master node?

Thanks again!

M

[martin_mueller]

Without clustering your minimum indexer count is 1.

Note, Splunk doesn't do active/passive of indexers - indexers in a cluster are all active. For detailed architecture discussions I recommend talking to a local Splunk Partner or your local Splunk Sales Engineer.

[martin_mueller]

An indexer cluster master cannot also be an indexer cluster peer.

[mshilston]

OK thanks, so that would be 7 servers in total then.

I assume if I didn't cluster the Indexes it would be 6?

[gjanders · Accepted solution]

The distributed deployment of Splunk would require a cluster master & indexers, to have a useful cluster you would want 2 nodes.
A search head could be standalone, so you could build your cluster with 3 servers.

If you want a search head cluster as well you need a deployer and 3 search heads at minimum!

I would suggest you build the cluster master (it's a very light server, it doesn't need a lot of CPU/memory) and then at least 2 indexers and 1 search head, this way you could survive an indexer outage by having duplicated data, you could not survive a search head outage though...

[hsesterhenn_spl]

I have to add some comments:

http://docs.splunk.com/Documentation/Splunk/6.5.3/Capacity/DimensionsofaSplunkEnterprisedeployment

• the easiest deployment (depending on the daily volume and number of concurrent users) is a single instance where search and indexing are combined

• if you get more data (>100 GB/day approx, depends) you might use "distributed search" where you usually have one search head and 2+ indexers (no HA!)
  http://docs.splunk.com/Documentation/Splunk/6.5.3/Deploy/Distributedoverview

• if you need HA and want to replicate your data in case of an outage, you'll need an "Indexer Cluster" with at least 2 Indexers, one Master Node (Cluster Master, might be a VM) and one Search Head.
  We are talking about 4 dedicated servers! (virtual or hardware).

• if you have a lot of concurrent users you might create a Search Head cluster which needs at least 3 Search Heads and a separate "Deployer" instance which might be a VM again.

To sum it up... if you have requirements regarding HA and load balancing and higher daily volume you might end up with at least 8 servers. Some of them can be "small" VMs but they need to run on dedicated servers.

Hope this helps.

And yes, you might get in contact with Splunk Professional Services or a certified Splunk Partner helping you with the right architecture.

Holger

[mshilston]

Thanks Holger,

I think initially I'm looking at the 4 server deployment above; Index Cluster with two nodes, one Master Node and one Search Head. This seems suitable for initial volumes and we can scale out later.

I will certainly look at engaging the professional services as I progress with this.

M

[gjanders]

Your deployer can also be a cluster master, and also a license master. The deployers role is quite passive normally...

You can even use your cluster master as a deployment server, but as your installation grows you might want a dedicated deployment server...

[hsesterhenn_spl]

Jepp... If you plan to go beyond 100GB soon, a small cluster will help keeping efforts low.

Happy splunking