Index/send logs generated while Splunk indexer is down?

[johns3]

When you use a syslog server like syslog-ng or the Splunk Universal Forwarder, what happens to the logs if the Splunk indexer is not available. How do you get the logs that were generated while the indexer was down or not reachable to be indexed? Can you set syslog-ng or the Universal Forwarder to keep track of logs that have and have not been sent off to the indexer? then when the indexer is back online all the logs generated while it was off will be sent so you don't miss anything?

[johns3]

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

[johns3]

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

[chris]

Hi

The Universal Forwarder will fill its queues and the wait until the indexer becomes available again, with syslog(-ng) the recommended solution is to write the data to files on the receiving side and the index those files with splunk.

Have a look at:
http://splunk-base.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-...
http://splunk-base.splunk.com/answers/5216/how-to-send-syslog-ng-messages-to-splunk-properly

Chris

[dwaddle]

This is one of the reasons why I always recommend letting a syslog server handle the UDP input and spin syslog data to disk, instead of having Splunk listen on UDP directly. It gives you some resiliency in these type of events.

Note that monitored files are handled differently from scripted inputs and such. Splunk can successfully "pause" files (assuming they aren't archived underneath it), but data from scripted inputs will eventually be dropped once the queues fill up.

You might also want to look at the documentation area on "indexer acknowledgement"