universal forwarder: handling server unavailability

[wsw70]

Hello,

I have an environment with many machines sending syslog (udp) to a central indexer. Since the machines are sometimes far away from the indexer I wanted to build a more robust topology. My initial idea was to set up local rsyslog servers receiving on UDP and forwarding on TCP with the failover capacity built into rsyslog.

Unfortunately this does not work as expected (if someone is interested I will give some info at the end of the post). Then I realized that splunk has forwarders 🙂

My questions would be:

• is the forwarding done over TCP or though a mechanism which handles the possible unavailability of the machine data is forwarded to?
• in case the centralized indexer is not available, will the forwarder buffer the data and send it when the indexer is back online?

PS. As for the failed rsyslog scenario, I wanted to use the follwing configuration in /etc/rsyslog.conf:

:fromhost-ip, !isequal, "127.0.0.1" @@indexingserver.example.com
$ActionExecOnlyWhenPreviousIsSuspended on
& /var/log/buffer_indexingserver_514_TCP_not_available
$ActionExecOnlyWhenPreviousIsSuspended off

This works but the events gathered while indexingserver is offline are not forwarded, they stay in /var/log/buffer_indexingserver_514_TCP_not_available. I could imagine a way to index them though NFS mounted files but it gets complicated -- hopefully the splunk forwarder will be the ideal solution.

PPS. As a follow-up: nxlog does the above job correctly. I am still hoping to get some feedback about the splunk forwarders as a native, multiplatform solution would be best.

[gkanapathy · Accepted solution]

Yes. The Splunk forwarders send over TCP. Optionally with version 4.2 and higher at the cost of some performance, they can also perform end-to-end acknowledgment and wait until the data has been written to disk before committing (for file monitoring). It will also buffer some data in-memory, and optionally can persist to disk.

What I would recommend actually is that you receive the UDP data using rsyslog, then write it to local disk (split over files by host, if that's convenient). Set up log rotation on these local disk files to store however much you want buffered (1 day, 1 hours, whatever), and clean them once it's done. Then set up the Splunk forwarder to monitor and forward those files. Splunk will then watch the files, send data as is possible, and wait while keeping a pointer to the appropriate file location when it's not possible to send.

[wsw70]

Thanks for the detailed info. Since I am in a multi-OS scenario (Linux and MS Windows) using the forwarders alone would be the best solution (ie not having to install a syslog server on Windows). I will give it a try.

[Damien_Dallimor]

Just to augment what GK wrote...you may also wish to consider adding more Splunk Indexers for High Availability and Failover scenarios...the Universal Forwarder can then load balance over the Indexers.

Have a look at the outputs.conf spec for more details :
http://docs.splunk.com/Documentation/Splunk/latest/admin/Outputsconf

[alapour]

I don't know if this quite answers your question or not. But based on experience with Splunk, the forwarders have always "caught up" when our indexer may have been unavailable. In fact, I have the forwarder installed on mobile systems that when connected to our network send all their logs.