Persistent queues tcp

[lisaac]

Splunk has an option of a disk based persistent queue on a TCP input. The option is not available for splunktcp inputs. I have 10-20K hosts with a UF, and the UFs send into an HF layer (17 hosts running 2 HFs each). We do this to route data to hadoop, splunk enterprise, or the null bucket. The splunk version is 6.4.4 for the HFs.

I was thinking of updating my UFs to not send CookedData into the HF layer and change the Splunktcp inputs to TCP inputs with disk based persistent queues. Here is the setting from the outputs.conf.spec file:

sendCookedData = [true|false]
* If true, events are cooked (have been processed by Splunk).
* If false, events are raw and untouched prior to sending.
* Set to false if you are sending to a third-party system.
* Defaults to true.

The goal is to gain the benefit of Persistent queuing to store data in an input queue to disk. This can help prevent data loss if the forwarder or indexer gets backed up. Does anyone have any thoughts on this item? There will be extra overhead on the HF layer due to uncooked data. I would overcome this by using a combination of Splunktcp and tcp inputs. TCP inputs would handle any data item that would need an extra level of data loss prevention.

I have not tried increasing the input queue memory size. This could be an alternate approach to disk based queues.

[lguinn2]

Personally, I don't think that you need this. If you want to do data loss prevention
1: Turn on the useAck option on every forwarder
2: Make sure that you are not overloading the middle forwarder tier
3: Size the overall forwarder queue appropriately (maxQueueSize) - this is especially necessary on the middle forwarder tier

Finally, you should not be using heavy forwarders as a middle tier unless you absolutely must. Universal forwarders are preferred as they are more efficient. I know that is counter-intuitive for most people, but it is true. Here is a very recent blog post that addresses this: Universal or Heavy, that is the question