Getting Data In

Index/send logs generated while Splunk indexer is down?

johns3
Path Finder

When you use a syslog server like syslog-ng or the Splunk Universal Forwarder, what happens to the logs if the Splunk indexer is not available. How do you get the logs that were generated while the indexer was down or not reachable to be indexed? Can you set syslog-ng or the Universal Forwarder to keep track of logs that have and have not been sent off to the indexer? then when the indexer is back online all the logs generated while it was off will be sent so you don't miss anything?

Tags (1)

johns3
Path Finder

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

0 Karma

johns3
Path Finder

Thanks for the answers. So I should just log to syslog-ng and forward the files with the Linux universal forwarder?

0 Karma

chris
Motivator

Hi

The Universal Forwarder will fill its queues and the wait until the indexer becomes available again, with syslog(-ng) the recommended solution is to write the data to files on the receiving side and the index those files with splunk.

Have a look at:
http://splunk-base.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-...
http://splunk-base.splunk.com/answers/5216/how-to-send-syslog-ng-messages-to-splunk-properly

Chris

dwaddle
SplunkTrust
SplunkTrust

This is one of the reasons why I always recommend letting a syslog server handle the UDP input and spin syslog data to disk, instead of having Splunk listen on UDP directly. It gives you some resiliency in these type of events.

Note that monitored files are handled differently from scripted inputs and such. Splunk can successfully "pause" files (assuming they aren't archived underneath it), but data from scripted inputs will eventually be dropped once the queues fill up.

You might also want to look at the documentation area on "indexer acknowledgement"

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...