Universal Forwarder and Offline Scenarios

[helge]

I am using Universal Forwarder on Windows machines to forward events generated by a script.

Question: What happens if the indexer(s) cannot be reached for some reason? Are the events buffered locally and sent when the connection is restored? Is the behavior in this situation documented anywhere?

[okrabbe_splunk · Accepted solution]

To guard against loss of data when forwarding to an indexer, you can use Splunk's indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data not acknowledged as "received" by the indexer.

This topic is discussed at length in the documentation link below.

Protect against loss of in-flight data

About queues :

Memory queues, used by default. A forwarder crashing/shutdown while queueing data will lose the data waiting in the memory queue.

Persistent queues, you can configure your forwarders to keep part of the data queued on disk instead of memory, they will be resumed once restarted.

see http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

Also, this Splunk-base answer has some additional details:
http://splunk-base.splunk.com/answers/46073/does-the-indexer-acknowledgement-queuelist-persist-acros...

[okrabbe_splunk]

helge, I updated the answer to discuss using persistent queues. Hopefully this is what you were looking for.

[helge]

Thanks, that is helpful. However, the document does not discuss how much data the forwarder still accepts when it cannot reach an indexer (with acks enabled). Will it stop accepting once the queue is full? Or will it accept more? In that case, where is the data stored - in memory, or on disk?

[jagadeeshm]

I think 25MB - http://docs.splunk.com/Documentation/Splunk/6.4.4/Forwarding/Protectagainstlossofin-flightdata