Hello,
I am implementing Splunk.
1 Search Head
An indexer cluster with 2 peers
1 Master Node
X Heavy Forwarders
I have to deploy them across 2 datacenters.
Which is the best way to distribute these objects o the datacenters?
Thank you very much.
Why would you need that? Your source system forwarders will have all indexers configured in outputs.conf (or use indexer discovery), so if one goes down, they will continue to send data to the remaining two indexers.
Assuming you have network connectivity.
And if they can't, they will stop monitoring files until an indexer becomes reachable again.
I may have understood wrong.
The data sources have to send the events to all the peers in the cluster?
Won't that increase the license usage?
I thought that data sources send to only one indexer and then the peers replicate between themselves.
Yes, they send to one indexer at a time. The forwarders will auto load-balance across all available indexers configured in outputs.conf. It one of them is not available, they will pick another one.
Okok.
So answering" why would I do that".
It is because as the sources are in different datacenters, sources in datacenters A will send to hf in datacenter A which will send to indexer in datacenter A. Unless indexer A is down so IN THAT CASE the HF A should send to indexer in datacenter B.
I am seeing that the point of failure still exists in the HF right? If a HF is down I will be loosing events.
How can I solve it?
Thank you very much
First of all: If you have a UF monitor files and send to your HF, and that HF is not available, you will not lose data. The UF's will stop monitoring. Note that the story is different for TCP inputs, but that's a different story.
Secondly, yes, if you only have one intermediary HF, you introduced a single point of failure. Solution: Have at least two.
Thirdly: Back to my original recommendation: Remove the HF and do your event filtering on the indexers directly and have at least two indexers in each datacenter (use the HF server you just got back).
And lastly: If you want forwarders in datacenterA to failover to datacenterB, set up a multi-site cluster, use indexer discovery and configure forwarder site failover capability.
That will cause DatacenterA forwarders to send to indexers in the same datacenter, unless they all go down, in which case they failover to the other datacenter.
Thank you.
But as I told you I don't have any universal forwarder. And I won't be able to install any.
Thanks again
Where is your data coming from then? Is your HF listening form some network input? Monitoring files?
At this moment there is no HF either.
Data is transmitted directly to the standalone Splunk via syslog, monitoring files, network inputs
Thanks again
Maybe review the documentation here.
Regarding my comment about retaining a healthy cluster in the event of a peer failure: If you setup a cluster with two peers and ask it to maintain two copies of each bucket, the cluster will not be able to do that anymore if you lose a peer (because there is only one indexer left). You need at least three indexers to remain in healthy (fully replicated, fully searchable) state during a peer failure. Probably OK for your use case.
So if I include 3 indexers in the cluster, which is the Search Factor and Replication Factor recommendations?
Thank's again.
I would choose RF=2, SF=2 given what I understand about your requirements.