Getting Data In

Index/send logs generated while Splunk indexer is down?

johns3
Path Finder

When you use a syslog server like syslog-ng or the Splunk Universal Forwarder, what happens to the logs if the Splunk indexer is not available. How do you get the logs that were generated while the indexer was down or not reachable to be indexed? Can you set syslog-ng or the Universal Forwarder to keep track of logs that have and have not been sent off to the indexer? then when the indexer is back online all the logs generated while it was off will be sent so you don't miss anything?

Tags (1)

johns3
Path Finder

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

0 Karma

johns3
Path Finder

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

0 Karma

chris
Motivator

Hi

The Universal Forwarder will fill its queues and the wait until the indexer becomes available again, with syslog(-ng) the recommended solution is to write the data to files on the receiving side and the index those files with splunk.

Have a look at:
http://splunk-base.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-...
http://splunk-base.splunk.com/answers/5216/how-to-send-syslog-ng-messages-to-splunk-properly

Chris

dwaddle
SplunkTrust
SplunkTrust

This is one of the reasons why I always recommend letting a syslog server handle the UDP input and spin syslog data to disk, instead of having Splunk listen on UDP directly. It gives you some resiliency in these type of events.

Note that monitored files are handled differently from scripted inputs and such. Splunk can successfully "pause" files (assuming they aren't archived underneath it), but data from scripted inputs will eventually be dropped once the queues fill up.

You might also want to look at the documentation area on "indexer acknowledgement"

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...