Universal Forwarder buffering during Indexer outage?

[brettw10]

Say I have a UF set up to monitor a file – let’s call it /var/log/syslog-stats.log – which rotates every day (syslog-stats.log, syslog-stats.log.0, syslog-stats.log.1, etc.). The UF is only configured to look at syslog-stats.log, and not the rotated files.

Now let’s say that the Indexer that the UF forwards to goes down for a period of time. During this time, the syslog-stats.log file rotates. Once the Indexer comes on line again, will it only see the events that are in the current syslog-stats.log file, or will it also see the events that occurred while it was down, which are now recorded in syslog-stats.log.0?

I guess the question really is, at a much simpler level – will the UF buffer for a given period of time if an indexer is down? If so, how long for?

[lukejadamec]

If you use useACK=true in outputs.conf you can configure the buffer as you need.
useACK is used to verify that data sent to the indexer is received by the indexer. If the forwarder does not get an ACK, the the forwarder will resend the data when the indexer comes back on-line. You will need to configure the buffer to suit your needs, because when the buffer is full, then the forwarder will stop and wait - so you would loose logs that have both not been buffered and rotated.

The problem with useACK=true is that if the ACK message is lost (from the indexer to the forwarder) then the forwarder will resend the event, which will result in a duplicate of an event. Note: Everytime the forwarder resends the event it will record a 'possible duplication of event' entry in the splunkd log.