How to achieve The High availability and Disaster recovery architecture in Splunk distributed environment.

[Gk7]

Do we have any facility in the Splunk that we can achieve the High availability or Disaster recovery features in the Splunk. if yes, please share the documents for this. 

Your response will be appreciated.!!!

[richgalloway · Accepted solution]

Splunk has features that increase availability, but I would not call it an HA product.  Those features are:

1) Multi-site indexer cluster.  See https://docs.splunk.com/Documentation/Splunk/9.0.5/Indexer/Multisitearchitecture

2) Search head clustering.  See https://docs.splunk.com/Documentation/Splunk/9.0.5/DistSearch/SHCarchitecture

3) Indexer cluster manager redundancy.  See http://docs.splunk.com/Documentation/Splunk/9.0.5/Indexer/CMredundancy

See the Splunk Validated Architectures document (https://www.splunk.com/en_us/pdfs/tech-brief/splunk-validated-architectures.pdf), specifically architecture M4/M14.

[sonishar]

How can we do  High availability for Heavy Forwarders and SC4S

[PickleRick]

With HF - it can be complicated because the problem here typically would be not to have multiple instances but to _not_ have multiple input instances running at the same time and you'd need to replicate the state of the inputs in case of a need for fail-over. There is nothing out-of-the-box to do it. You can to devise something with zip ties and duct tape but those solutions typically have some issues specific to chosen architecture.

Of course if you're not running any scripted/modular inputs and only have HFs as a "parsing layer" in front of indexes, there is no problem with having multiple HFs receiving data from UFs.

With SC4S there is no problem with running multiple instances. The problem is that you want the sources to send only to one of them. You can try to do some tricks with "floating IP" either on the hosts themselves using keepalived or something similar or on the router using some form of network-level load-balancing but it doesn't give you 100% guarantee of no data loss during the switchover period. It's just how the syslog works.

[Gk7]

Ya done now. 

[isoutamo]

As @richgalloway already pointed you could do some kind of HA system with splunk. Indexing tier is real HA with multi site cluster, but SH tier didn’t. With SHC you could get better availability, but you should remember that it’s not designed as a HA!

[PickleRick]

Hmm. That's interesting.

I don't want to challenge your opinion. I'm just curious as to why you both don't treat SHC as a highly-available solution. I'd say it ticks all the boxes.