Getting Data In

Universal Forwarder and Offline Scenarios

helge
Builder

I am using Universal Forwarder on Windows machines to forward events generated by a script.

Question: What happens if the indexer(s) cannot be reached for some reason? Are the events buffered locally and sent when the connection is restored? Is the behavior in this situation documented anywhere?

Tags (3)
1 Solution

okrabbe_splunk
Splunk Employee
Splunk Employee

To guard against loss of data when forwarding to an indexer, you can use Splunk's indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data not acknowledged as "received" by the indexer.

This topic is discussed at length in the documentation link below.

Protect against loss of in-flight data

About queues :

Memory queues, used by default. A forwarder crashing/shutdown while queueing data will lose the data waiting in the memory queue.

Persistent queues, you can configure your forwarders to keep part of the data queued on disk instead of memory, they will be resumed once restarted.

see http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

Also, this Splunk-base answer has some additional details:
http://splunk-base.splunk.com/answers/46073/does-the-indexer-acknowledgement-queuelist-persist-acros...

View solution in original post

okrabbe_splunk
Splunk Employee
Splunk Employee

To guard against loss of data when forwarding to an indexer, you can use Splunk's indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data not acknowledged as "received" by the indexer.

This topic is discussed at length in the documentation link below.

Protect against loss of in-flight data

About queues :

Memory queues, used by default. A forwarder crashing/shutdown while queueing data will lose the data waiting in the memory queue.

Persistent queues, you can configure your forwarders to keep part of the data queued on disk instead of memory, they will be resumed once restarted.

see http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

Also, this Splunk-base answer has some additional details:
http://splunk-base.splunk.com/answers/46073/does-the-indexer-acknowledgement-queuelist-persist-acros...

okrabbe_splunk
Splunk Employee
Splunk Employee

helge, I updated the answer to discuss using persistent queues. Hopefully this is what you were looking for.

0 Karma

helge
Builder

Thanks, that is helpful. However, the document does not discuss how much data the forwarder still accepts when it cannot reach an indexer (with acks enabled). Will it stop accepting once the queue is full? Or will it accept more? In that case, where is the data stored - in memory, or on disk?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...