Add-ons on multiple Heavy Forwarders for HighAvailability ?

[Mosstrow]

Hello,

I have a distributed environment in which there are a cluster of indexers, 3 heavy forwarders and 3 search head

How do you guys managed the high availability of add-on/TA on HF clusters ?

By example I would like to install AWS Add-on to get data by REST API:

- If I configure TA on 3 HF I'll get 3 times same data. (Indexed 3 times, my licenses will explode)

- If I configure TA on only one HF I'd have problems with high availability in case of this node have failure.

Thanks

[Mosstrow]

Hi @gcusello  @PickleRick,

Thanks for the quick reply.

Correct, our 3 HF receive data from syslogs behind a F5 loadbalancer, they are all in active/active mode and the loadbalancer distribute data on each HF. These HF should be use also to pull data to various cloud.

If I understand well splunk does not propose a solution out of the box for this case.

As our Search-Head are in cluster mode I could maybe continuously searching HF failure in logs (F5 logs) and configure an alert with a script to enable the TA with API call or whatever on another HF in case of failure.
In that case we also need to configure rsync or other to copy checksum to other HF node to don't pull all data again.
I may overthinking and complexify too much

[gcusello]

Hi @Mosstrow,

for the data that you're reciving on HF through the F5, you don't need to do nothing because it's the F5 that manages the failure redirecting logs to another HF.

For the Pull HF, Yes, you could create an alert that start a script to enable the TA on another HF.

Only one thing: you can search an eventual failure on Splunk internal logs (_internal) of the Pull HF instead on F5, but it's the same thing.

Ciao.

Giuseppe

[PickleRick]

Well, if you pull the events it's up to you to organize it so that only one "puller" works at the time. If you have push-orientated log transport (i.e. syslog), it's also up to you but it's easier because you can do that on network level (multiple receivers and floating IP or even a network-level load balancing). But with pull-based transport you can also try some tricks. For example - floating IP and source-based access control on the device you're pulling logs from. Ugly, but possible.

[gcusello]

Hi @Mosstrow,

if you have an HA feature (e.g. VM-Ware) you don't need to have two HFs because the HF  with REST API doesn't receive data it asks data to AWS and receive them.

This means that you don't lose data in case of fault.

It's different if you have HFs that receive data from syslogs or other Forwarders.

If you haven't an HA feature, you could use a policy of active - passive, installing the TA on two HFs, activating only one of them at one time.

Ciao.

Giuseppe