I want to configure usePersistentQueue so Splunk will buffer my syslog events.

[the_wolverine]

What are the gotchas to configuring this other than what is documented in this Splunk Wiki article which states that this should NOT be configured in version 4.0 and later except for very specific use cases (contact Support)?

[bmignosa_splunk · Accepted solution]

Persistent Queues are available for use again as of 4.2 and newer.
http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

[the_wolverine]

In version 4.0, the general consensus is DO NOT USE usePersistentQueue. Further, Splunk has documented that the feature is deprecated. The default behavior (in 4.0) is that, when a forwarder loses its connection to the indexer for whatever reason, it will block continuously instead of writing to disk. For UDP/syslog inputs, this means any events forwarded during the time that queues are blocked will be dropped.

If the concern is prevention of data loss, you should consider going to TCP input and using flat files (e.g. syslog-ng).

Be forewarned that a usePersistentQueue-enabled configuration will require very careful planning as it is heavily dependent on hardware resources. You will be responsible for fully testing this configuration before Production implementation. If you still choose to go this route, be aware of the scenario where, if queues are blocked and the Splunk forwarder is writing to disk, you cannot restart the forwarder until the "issue" is resolved or events will be lost.