tcp and persistent queues

[fernandoandre]

After reading this and this I'm not sure about the use of persistent queues on Splunk.

In particular, in one implementation I'm involved with, we have one Heavy Forwarder that aggregates all Universal Forwarders connections and all syslog connections and then forward data to Indexer. In fact one of the goals of having a HF is to make cache in the case the Indexer fails for some reason. This could be accomplished with the use of persistent queues on the HF.

Therefore my question is, should I use persistent queuing with splunktcp or not? Is it recommended or is anyone using it and have tested it?

Another question relates to the use of the attribute "persistentQueueSize". I'm using it as the following piece exemplifies but Splunk outputs the following:

"Possible typo in stanza [splunktcp://:9997] in /opt/splunk/etc/system/local/inputs.conf,: persistentQueueSize = 10GB". What is the problem here? I've copied it exactly as it is in inputs.conf example on splunk documentation.

[splunktcp://:9997]
connection_host = ip
_TCP_ROUTING = splunkssl
persistentQueueSize = 10GB

Thank you in advance.

[fernandoandre · Accepted solution]

I have done some experiments with this feature and I can confirm that works perfectly in Splunk (Heavy Forwarder) 4.3.x

There is this file that Splunk Heavy Forwarder creates on disk and keeps growing it until the HF is able to connect again to the Indexer and send him the data, where obviously the file size decreases.

[mikaelbje]

I'm just wondering if this still is a working solution, as the official documentation states it won't work. From the documentation at http://docs.splunk.com/Documentation/Splunk/6.1.4/Data/Usepersistentqueues:

Persistent queues are available for these input types:

• TCP
• UDP
• FIFO
• Scripted inputs
• Windows Event Log inputs

Persistent queues are not available for these input types:

• Monitor
• Batch
• File system change monitor
• splunktcp (input from Splunk forwarders)

I have a scenario where I'd like to put intermediate forwarders (also acting as deployment servers) in different security zones to limit the traffic flow between zones. In an event where the indexers go down I need to have the intermediate forwarder buffer data from universal forwarders on disk. We have an additional complication too, and that is the one of useACK=true. Obviously an acknowledgement from the indexer cannot be sent if it's down, and it's not the intermediate forwarder's job to acknowledge, so it sounds like a catch-22 to me. Is the only option to increase the buffers or persistentQueueSize on the universal forwarders and let them handle all the buffering?

My scenario is in a way similar to the one in http://answers.splunk.com/answers/78388/splunk-store-and-forward-ha.html