Deployment Architecture

Does it make sense to have a DR indexer be part of cluster?

HathMH
Path Finder

I inherited a splunk mesh of search-heads, deployment server, index cluster, etc. I am trying to figure out all this splunk stuff, but ran into an issue that I am not sure if it ignores best practice, poor judgement, or as intended.

We have 8 main indexers that do what indexers do, all clustered as peer nodes.

The deployment server is the master node and the search head for the cluster (which I don't understand that since we also have 5 main separate search heads). 

We also have a disaster recovery DR site that has an indexer as a peer node part of the aforementioned cluster.

The cluster has a Replication factor 3, for # of copies of raw data.

The cluster has a Search factor of 2, for # of searchable copies.

Newer to cyber so forgive me if I don't understand right away or if I am missing something glaringly obvious. But does it makes sense to have the DR indexer be part of the cluster? If it does makes sense, then how do i ensure that the other 8 indexers send a copy of all their stuff to the DR indexer? I thought the master node just kind of juggles the incoming streams from the forwarders and balances the data across all the indexers. 

Also;

- should the deployment server double as a master node and search head for the index cluster?

- what is the difference between the 5 main separate search heads and the search head in the index cluster?

- (last one, i swear) would it make sense to have a search head cluster, or keep the search heads separate as they the 5 are accessed and used by different groups (networking, development, QA/testing, cybersecurity, and UBA (which we dont even have UBA servers active right now cuz I cannot get them to work or web ui to launch))

Labels (2)
0 Karma
1 Solution

PickleRick
Ultra Champion

I don't understand your "DR" architecture.

Normally with Splunk to have high availability you create a multisite cluster with properly set up site RF and SF.

That way the buckets are automatically replicated across the sites and enough number of copies are kept both within a single site as well as across your sites.

If you have a single site cluster with one of the clusters located in a different location than the rest of them, unless you have RF equal to number of your indexers (which is a very rare setup when you have more than 2 indexers), you will _not_ have a copy of each of your bucket in that "DR" site. That makes no sense.

In case of small deployments you sometimes use single node for multiple roles (deployment server, cluster master, deployer/monitoring console). Depends on the load. With few indexers/SHs/deployment clients you can get away with that. Often master is a separate search-head when doubled with monitoring console. It's just not used for typical "production" search workload.

Search head cluster and indexer cluster are two separate entities. So you don't have "search head in indexer cluster".

5 SHs is quite a lot so it's worth reviewing whether you want clustering or not. And whether you can have less SHs or need more of them. But that needs reviewing your  whole setup, its performance, load. And your organizational needs. So it's not that easy.

One thing is obvious however - managing a 5-node SH-cluster should be more straightforward than 5 separate ones. But - on the other hand - managing workload can be easier if - for example - each team is restricted to own SH.

So it's not that easy to give generalized answer here.

View solution in original post

PickleRick
Ultra Champion

I don't understand your "DR" architecture.

Normally with Splunk to have high availability you create a multisite cluster with properly set up site RF and SF.

That way the buckets are automatically replicated across the sites and enough number of copies are kept both within a single site as well as across your sites.

If you have a single site cluster with one of the clusters located in a different location than the rest of them, unless you have RF equal to number of your indexers (which is a very rare setup when you have more than 2 indexers), you will _not_ have a copy of each of your bucket in that "DR" site. That makes no sense.

In case of small deployments you sometimes use single node for multiple roles (deployment server, cluster master, deployer/monitoring console). Depends on the load. With few indexers/SHs/deployment clients you can get away with that. Often master is a separate search-head when doubled with monitoring console. It's just not used for typical "production" search workload.

Search head cluster and indexer cluster are two separate entities. So you don't have "search head in indexer cluster".

5 SHs is quite a lot so it's worth reviewing whether you want clustering or not. And whether you can have less SHs or need more of them. But that needs reviewing your  whole setup, its performance, load. And your organizational needs. So it's not that easy.

One thing is obvious however - managing a 5-node SH-cluster should be more straightforward than 5 separate ones. But - on the other hand - managing workload can be easier if - for example - each team is restricted to own SH.

So it's not that easy to give generalized answer here.

HathMH
Path Finder

When setting up a index cluster, is there an option to make it multisite or something? I ask because our index cluster has 8 indexers on our 192.x main network, 1 deployment/master node on the 192.x network, and then 1 indexer on the 10.x network (our DR disaster recovery site).

0 Karma

PickleRick
Ultra Champion

Yes, there is an option for a cluster to be multisite. You can also migrate from a single site cluster to multisite but it takes some work.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Depending on how busy your Splunk environment is, you could get away with having the same instance serve as both deployment server and cluster manager (I'm not sure what you mean by "search head for the index cluster).

Having a search head cluster (SHC) allows the nodes in the cluster to share resources.  That means you have more search "slots" for running scheduled searches so it's less likely a search will be skipped because there was no CPU available.  I'd keep it if it's all the same to your company.  Also, a SHC allows a search head to be taken down for maintenance while still allowing users access to their stuff.

I'm a bit concerned about your DR system, however.  Yes, the DR should be part of the indexer cluster.  That allows it to receive replicated buckets from other indexers.  My concern is about how a single DR indexer could possibly take the load of the other 8 indexers.  Or am I misunderstanding and the DR indexer is there in case one of the other 8 fails?  If that's the case then it's not DR - it's just part of the cluster.  All members of an indexer cluster help when one of the peers fails.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...