Does it make sense to have a DR indexer be part of cluster?

[HathMH]

I inherited a splunk mesh of search-heads, deployment server, index cluster, etc. I am trying to figure out all this splunk stuff, but ran into an issue that I am not sure if it ignores best practice, poor judgement, or as intended.

We have 8 main indexers that do what indexers do, all clustered as peer nodes.

The deployment server is the master node and the search head for the cluster (which I don't understand that since we also have 5 main separate search heads). 

We also have a disaster recovery DR site that has an indexer as a peer node part of the aforementioned cluster.

The cluster has a Replication factor 3, for # of copies of raw data.

The cluster has a Search factor of 2, for # of searchable copies.

Newer to cyber so forgive me if I don't understand right away or if I am missing something glaringly obvious. But does it makes sense to have the DR indexer be part of the cluster? If it does makes sense, then how do i ensure that the other 8 indexers send a copy of all their stuff to the DR indexer? I thought the master node just kind of juggles the incoming streams from the forwarders and balances the data across all the indexers. 

Also;

- should the deployment server double as a master node and search head for the index cluster?

- what is the difference between the 5 main separate search heads and the search head in the index cluster?

- (last one, i swear) would it make sense to have a search head cluster, or keep the search heads separate as they the 5 are accessed and used by different groups (networking, development, QA/testing, cybersecurity, and UBA (which we dont even have UBA servers active right now cuz I cannot get them to work or web ui to launch))

[PickleRick · Accepted solution]

I don't understand your "DR" architecture.

Normally with Splunk to have high availability you create a multisite cluster with properly set up site RF and SF.

That way the buckets are automatically replicated across the sites and enough number of copies are kept both within a single site as well as across your sites.

If you have a single site cluster with one of the clusters located in a different location than the rest of them, unless you have RF equal to number of your indexers (which is a very rare setup when you have more than 2 indexers), you will _not_ have a copy of each of your bucket in that "DR" site. That makes no sense.

In case of small deployments you sometimes use single node for multiple roles (deployment server, cluster master, deployer/monitoring console). Depends on the load. With few indexers/SHs/deployment clients you can get away with that. Often master is a separate search-head when doubled with monitoring console. It's just not used for typical "production" search workload.

Search head cluster and indexer cluster are two separate entities. So you don't have "search head in indexer cluster".

5 SHs is quite a lot so it's worth reviewing whether you want clustering or not. And whether you can have less SHs or need more of them. But that needs reviewing your  whole setup, its performance, load. And your organizational needs. So it's not that easy.

One thing is obvious however - managing a 5-node SH-cluster should be more straightforward than 5 separate ones. But - on the other hand - managing workload can be easier if - for example - each team is restricted to own SH.

So it's not that easy to give generalized answer here.

[HathMH]

When setting up a index cluster, is there an option to make it multisite or something? I ask because our index cluster has 8 indexers on our 192.x main network, 1 deployment/master node on the 192.x network, and then 1 indexer on the 10.x network (our DR disaster recovery site).

[PickleRick]

Yes, there is an option for a cluster to be multisite. You can also migrate from a single site cluster to multisite but it takes some work.

[richgalloway]

Depending on how busy your Splunk environment is, you could get away with having the same instance serve as both deployment server and cluster manager (I'm not sure what you mean by "search head for the index cluster).

Having a search head cluster (SHC) allows the nodes in the cluster to share resources.  That means you have more search "slots" for running scheduled searches so it's less likely a search will be skipped because there was no CPU available.  I'd keep it if it's all the same to your company.  Also, a SHC allows a search head to be taken down for maintenance while still allowing users access to their stuff.

I'm a bit concerned about your DR system, however.  Yes, the DR should be part of the indexer cluster.  That allows it to receive replicated buckets from other indexers.  My concern is about how a single DR indexer could possibly take the load of the other 8 indexers.  Or am I misunderstanding and the DR indexer is there in case one of the other 8 fails?  If that's the case then it's not DR - it's just part of the cluster.  All members of an indexer cluster help when one of the peers fails.