Splunk Enterprise

indexer cluster topology across two datacenters

noybin
Communicator

Hello,

I am implementing Splunk.

1 Search Head
An indexer cluster with 2 peers
1 Master Node
X Heavy Forwarders

I have to deploy them across 2 datacenters.

Which is the best way to distribute these objects o the datacenters?

Thank you very much.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Why would you need that? Your source system forwarders will have all indexers configured in outputs.conf (or use indexer discovery), so if one goes down, they will continue to send data to the remaining two indexers.
Assuming you have network connectivity.
And if they can't, they will stop monitoring files until an indexer becomes reachable again.

0 Karma

noybin
Communicator

I may have understood wrong.

The data sources have to send the events to all the peers in the cluster?
Won't that increase the license usage?

I thought that data sources send to only one indexer and then the peers replicate between themselves.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Yes, they send to one indexer at a time. The forwarders will auto load-balance across all available indexers configured in outputs.conf. It one of them is not available, they will pick another one.

noybin
Communicator

Okok.

So answering" why would I do that".
It is because as the sources are in different datacenters, sources in datacenters A will send to hf in datacenter A which will send to indexer in datacenter A. Unless indexer A is down so IN THAT CASE the HF A should send to indexer in datacenter B.

I am seeing that the point of failure still exists in the HF right? If a HF is down I will be loosing events.

How can I solve it?

Thank you very much

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

First of all: If you have a UF monitor files and send to your HF, and that HF is not available, you will not lose data. The UF's will stop monitoring. Note that the story is different for TCP inputs, but that's a different story.

Secondly, yes, if you only have one intermediary HF, you introduced a single point of failure. Solution: Have at least two.

Thirdly: Back to my original recommendation: Remove the HF and do your event filtering on the indexers directly and have at least two indexers in each datacenter (use the HF server you just got back).

And lastly: If you want forwarders in datacenterA to failover to datacenterB, set up a multi-site cluster, use indexer discovery and configure forwarder site failover capability.
That will cause DatacenterA forwarders to send to indexers in the same datacenter, unless they all go down, in which case they failover to the other datacenter.

noybin
Communicator

Thank you.

But as I told you I don't have any universal forwarder. And I won't be able to install any.

Thanks again

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Where is your data coming from then? Is your HF listening form some network input? Monitoring files?

0 Karma

noybin
Communicator

At this moment there is no HF either.

Data is transmitted directly to the standalone Splunk via syslog, monitoring files, network inputs

Thanks again

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Maybe review the documentation here.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Regarding my comment about retaining a healthy cluster in the event of a peer failure: If you setup a cluster with two peers and ask it to maintain two copies of each bucket, the cluster will not be able to do that anymore if you lose a peer (because there is only one indexer left). You need at least three indexers to remain in healthy (fully replicated, fully searchable) state during a peer failure. Probably OK for your use case.

0 Karma

noybin
Communicator

So if I include 3 indexers in the cluster, which is the Search Factor and Replication Factor recommendations?

Thank's again.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

I would choose RF=2, SF=2 given what I understand about your requirements.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...