topic Re: indexer cluster topology across two datacenters in Splunk Enterprise

indexer cluster topology across two datacenters

noybin — Wed, 30 Aug 2017 14:55:40 GMT

Hello,

I am implementing Splunk.

1 Search Head
An indexer cluster with 2 peers
1 Master Node
X Heavy Forwarders

I have to deploy them across 2 datacenters.

Which is the best way to distribute these objects o the datacenters?

Thank you very much.

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 30 Aug 2017 18:07:04 GMT

There is way too little information here to give a credible answer.
- What are your requirements that drive "I have to deploy them across 2 data centers"? HA? DR? What?
- What is your daily data ingest volume
- What is the network latency between your data centers?
- What are the heavy forwarders for?

Note that an indexer cluster with just two peers cannot remain in healthy state if you lose a single indexer, you need at least three indexers to maintain an RF=2/SF=2 cluster in healthy state during peer outage.

But like I said, it would help to understand your design goals and constraints better, before giving you any kind of advice.

Re: indexer cluster topology across two datacenters

noybin — Fri, 01 Sep 2017 20:23:55 GMT

HA
5gb-7gb
0.5ms
Events Filtering.

I didn't understand what you said about: "Note that an indexer cluster with just two peers cannot remain in healthy state if you lose a single indexer, you need at least three indexers to maintain an RF=2/SF=2 cluster in healthy state during peer outage."

Thank you very much

Re: indexer cluster topology across two datacenters

s2_splunk — Fri, 01 Sep 2017 20:37:03 GMT

True HA cannot really be achieved with a single search head, so I am going to assume you need to ensure you can search your indexed data in case you have a data center outage.

The simplest is a two-node cluster, with one indexer in each data center; configured with RF=2/SF=2. This ensures that each indexer has a searchable copy of each bucket.
But that really only solves half of your problem, because you still need to have a failover plan for your search head and your cluster master node. Virtualization is your friend for these.

You don't need HFs to do event filtering, you can do that directly on the indexers just as well.

Re: indexer cluster topology across two datacenters

s2_splunk — Fri, 01 Sep 2017 20:39:34 GMT

Regarding my comment about retaining a healthy cluster in the event of a peer failure: If you setup a cluster with two peers and ask it to maintain two copies of each bucket, the cluster will not be able to do that anymore if you lose a peer (because there is only one indexer left). You need at least three indexers to remain in healthy (fully replicated, fully searchable) state during a peer failure. Probably OK for your use case.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 13:32:00 GMT

Thank you!

One last question.

Is it or is it not recommended to use Heavy Forwarders in this scenario.

I think about buffering, filtering maybe.

What is the best practice?

Thank you very much.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 13:40:46 GMT

So if I include 3 indexers in the cluster, which is the Search Factor and Replication Factor recommendations?

Thank's again.

Re: indexer cluster topology across two datacenters

jkat54 — Tue, 29 Sep 2020 15:39:32 GMT

I think you need 2 indexers in each data center for SF2/RF2 to be truly HA in the case of a data center going down. Also 2 search heads in each. So here's how I would do it:

Data Center A (Site1):
2 search heads
1 cluster master / search head deployer / deployment server
2 peers (indexers)

Data Center B (Site2):
2 search heads
1 standby (always offline unless needed) cluster master / search head deployer / deployment server
2 peers (indexers)

Multi-site Cluster Enabled
site_replication_factor = origin:1, site1:1, site2:1, total:2
site_search_factor = origin:2, site1:2, site2:2, total:4

The site_replication_factor guarantees there is at least 1 copy of the (non-searchable, but recoverable) buckets in each data center.

The site_search_factor guarantees there are at least 2 copies of the searchable buckets in each data center.

You storage will be calculated as follows:
Daily Ingestion Volume x 4 (SF) x 0.35 (SF Storage Ratio) + Daily Ingestion Volume x 2 (RF) x 0.15 (RF Storage Ration) * Retention in Days = Total Storage Needed

Divide that by 4 to get Total Storage Needed per Indexer

You will never achieve HA with the SHC and only 2 sites / data centers. This is because the SHC needs a majority vote to elect a new captain if the captain goes down. So for example if you have 4 search heads, 2 in site 1, 2 in site 2, and site 2 goes down... now you have just 2 of the original 4 search heads in Site 1 and they cant get a majority vote for who should be the captain. What you could do is put 3 search heads in site 1 and 2 search heads in site 2. Then if site 2 goes offline and one of its search heads was the captain, theres a majority of 3 search heads in site 1. Of course you may not need this level of HA, and are ok with having to manually elect a captain via CLI. In which case you could just put 2 in each data center.

As for the Cluster Master / Deployment Server / SHC Deployer... there's really not a super easy solution to make these HA.

You can put multiple deployment servers behind a load balancer and point all the forwarders to the load balancer VIP, but you cant do the same with the cluster master or the deployer without another layer of cloning / syncing (to be handled outside of Splunk's toolset). Typically VMware's vmotion is a good option for these devices.

Sorry to add this as an answer but we're running out of space for comments on the original question.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 14:53:54 GMT

Another one:

If I include a HF in each datacenter. And an indexer goes down. Can the HF automatically start sending the events to the indexer in the other Datacenter?

In that way I won't loose events when an indexer goes down in a datacenter.

Is that possible?
Is it recommended?

The problem about adding more indexers is that I will need to backup all of them increasing the cost of the backup.

Thank you very much.

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 17:37:11 GMT

I would choose RF=2, SF=2 given what I understand about your requirements.

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 17:39:16 GMT

Why would you need that? Your source system forwarders will have all indexers configured in outputs.conf (or use indexer discovery), so if one goes down, they will continue to send data to the remaining two indexers.
Assuming you have network connectivity.
And if they can't, they will stop monitoring files until an indexer becomes reachable again.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 17:44:31 GMT

I may have understood wrong.

The data sources have to send the events to all the peers in the cluster?
Won't that increase the license usage?

I thought that data sources send to only one indexer and then the peers replicate between themselves.

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 17:47:38 GMT

Yes, they send to one indexer at a time. The forwarders will auto load-balance across all available indexers configured in outputs.conf. It one of them is not available, they will pick another one.

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 17:49:23 GMT

Maybe review the documentation here.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 17:55:36 GMT

Okok.

So answering" why would I do that".
It is because as the sources are in different datacenters, sources in datacenters A will send to hf in datacenter A which will send to indexer in datacenter A. Unless indexer A is down so IN THAT CASE the HF A should send to indexer in datacenter B.

I am seeing that the point of failure still exists in the HF right? If a HF is down I will be loosing events.

How can I solve it?

Thank you very much

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 18:06:22 GMT

First of all: If you have a UF monitor files and send to your HF, and that HF is not available, you will not lose data. The UF's will stop monitoring. Note that the story is different for TCP inputs, but that's a different story.

Secondly, yes, if you only have one intermediary HF, you introduced a single point of failure. Solution: Have at least two.

Thirdly: Back to my original recommendation: Remove the HF and do your event filtering on the indexers directly and have at least two indexers in each datacenter (use the HF server you just got back).

And lastly: If you want forwarders in datacenterA to failover to datacenterB, set up a multi-site cluster, use indexer discovery and configure forwarder site failover capability.
That will cause DatacenterA forwarders to send to indexers in the same datacenter, unless they all go down, in which case they failover to the other datacenter.

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 19:14:04 GMT

Thank you.

But as I told you I don't have any universal forwarder. And I won't be able to install any.

Thanks again

Re: indexer cluster topology across two datacenters

s2_splunk — Wed, 06 Sep 2017 19:20:08 GMT

Where is your data coming from then? Is your HF listening form some network input? Monitoring files?

Re: indexer cluster topology across two datacenters

noybin — Wed, 06 Sep 2017 19:32:57 GMT

At this moment there is no HF either.

Data is transmitted directly to the standalone Splunk via syslog, monitoring files, network inputs

Thanks again

Re: indexer cluster topology across two datacenters

noybin — Thu, 07 Sep 2017 00:23:48 GMT

Thanks for your reply.

The client wants HA on indexing layer, not Search layer.

So we are creating indexer cluster. And of possible Heavy Forwarder cluster

Is it possible to cluster the HF?

Thank you very much