All Apps and Add-ons

Add-ons on multiple Heavy Forwarders for HighAvailability ?

Mosstrow
New Member

Hello,

I have a distributed environment in which there are a cluster of indexers, 3 heavy forwarders and 3 search head

How do you guys managed the high availability of add-on/TA on HF clusters ?

By example I would like to install AWS Add-on to get data by REST API:

- If I configure TA on 3 HF I'll get 3 times same data. (Indexed 3 times, my licenses will explode)

- If I configure TA on only one HF I'd have problems with high availability in case of this node have failure.

Thanks

 

 

 

Labels (1)
0 Karma

Mosstrow
New Member

Hi @gcusello  @PickleRick,

Thanks for the quick reply.

Correct, our 3 HF receive data from syslogs behind a F5 loadbalancer, they are all in active/active mode and the loadbalancer distribute data on each HF. These HF should be use also to pull data to various cloud.

If I understand well splunk does not propose a solution out of the box for this case.

As our Search-Head are in cluster mode I could maybe continuously searching HF failure in logs (F5 logs) and configure an alert with a script to enable the TA with API call or whatever on another HF in case of failure.
In that case we also need to configure rsync or other to copy checksum to other HF node to don't pull all data again.
I may overthinking and complexify too much

 

 

 

0 Karma

gcusello
Legend

Hi @Mosstrow,

for the data that you're reciving on HF through the F5, you don't need to do nothing because it's the F5 that manages the failure redirecting logs to another HF.

For the Pull HF, Yes, you could create an alert that start a script to enable the TA on another HF.

Only one thing: you can search an eventual failure on Splunk internal logs (_internal) of the Pull HF instead on F5, but it's the same thing.

Ciao.

Giuseppe

0 Karma

PickleRick
Ultra Champion

Well, if you pull the events it's up to you to organize it so that only one "puller" works at the time. If you have push-orientated log transport (i.e. syslog), it's also up to you but it's easier because you can do that on network level (multiple receivers and floating IP or even a network-level load balancing). But with pull-based transport you can also try some tricks. For example - floating IP and source-based access control on the device you're pulling logs from. Ugly, but possible.

0 Karma

gcusello
Legend

Hi @Mosstrow,

if you have an HA feature (e.g. VM-Ware) you don't need to have two HFs because the HF  with REST API doesn't receive data it asks data to AWS and receive them.

This means that you don't lose data in case of fault.

It's different if you have HFs that receive data from syslogs or other Forwarders.

If you haven't an HA feature, you could use a policy of active - passive, installing the TA on two HFs, activating only one of them at one time.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...