What are suggested approaches/steps for Syslog to switch to another heavy forwarder when primary goes down?

[neelamssantosh]

Hi Splunkers,

Kindly suggest any approach/steps for

Syslog must Switch to another Splunk Heavy Forwarder automatically when One Splunk Heavy Forwarder goes down.

If Primary FWD goes then it must switch to Secondary FWD.

Thanks in Advance.

[hortonew · Accepted solution]

Depends on your infrastructure, but one way which we use is to set up a virtual IP on a network device that you send all log data to, which can then map to multiple physical addresses. If one of the physical addresses in the pool goes down, it'll still send to the ones remaining.

[musskopf · Accepted solution]

Here my 2cents:

HeavyForwarders don't support it. As alternative solution I suggest one of below:

1. Use the syslog destination as a FQN A/CNAME in the DNS and update it if you need to failover - Might take some time til the DNS changes to replicate

2. Use a Virtual IP and a Cluster software, something like Linux heartbeat

3. If you have a device like F5 in your network, configure the virtual IP and fail-over rules there

4. Send syslog to both Splunk and perform a Dedup before index the data - waste of bandwidth/load?

5. Send syslog to both Splunk but one of the destinations will be off-line or blocked via firewall - you might have duplicated data if you end-up with both up at same time. Also similar bandwidth issue as above.

I'm probably going to have a F5 device in front of my heavy forwarders. The syslogs messages will be send via UDP from the devices but the F5 check rule will be checking the TCP/514 to confirm the service is up. I believe F5 will also allow me to create affinity rules....

Let me know if you have any other idea... I have similar problem with DBX App, that I still finding a better solution instead o manually migrate the configuration in case of failure.

[kylerose]

With regards to #1, I changed the A record in DNS and splunk never resolves it again. It requires a reboot to updates resolution. With tcpout, you can set dnsresolutioninterval , but not with syslog:

Invalid key in stanza [syslog:mysyslog] in /opt/splunk/etc/system/local/outputs.conf, line 13: dnsresolutioninterval (value: 300)

[neelamssantosh]

Kindly correct me if i am wrong,
can i achieve it with check sum redundancy in inputs.conf
crcSalt = <-source>

As perimeter device(Syslog) can send syslog logs to up-to 4IP's/HFW's.

it avoids the duplicate data while Re-indexing.

[neelamssantosh]

Thanks MuS,
For quick response.

Cheers..!!

[MuS]

Nice list of options 😉

Just one more to add: Linux Heartbeat can do this too

But remember, this Question is not related to Splunk. This feature must be configured outside Splunk.

cheers, MuS

[hortonew · Accepted solution]

Depends on your infrastructure, but one way which we use is to set up a virtual IP on a network device that you send all log data to, which can then map to multiple physical addresses. If one of the physical addresses in the pool goes down, it'll still send to the ones remaining.

[grijhwani]

You're missing what he said. You have an intervening device presenting a static virtual IP, which (as with the F5 load-balancer suggested in the previous answer) monitors the health of the various available forwarders, and routes the traffic accordingly.

Remember, by default syslog is UDP - i.e. stateless. As remarked, something needs to be present on the indexer to indicate the health of the syslog inbound on each host.

[neelamssantosh]

Thanks hortonew,
Syslog device server can send upto 4IP Address in parallel for High Availability of Forwarders but thats not the case we are looking for.
If Primary FWD goes then it must switch to Secondary FWD.