Splunk Search

Persistent queues tcp

lisaac
Path Finder

Splunk has an option of a disk based persistent queue on a TCP input. The option is not available for splunktcp inputs. I have 10-20K hosts with a UF, and the UFs send into an HF layer (17 hosts running 2 HFs each). We do this to route data to hadoop, splunk enterprise, or the null bucket. The splunk version is 6.4.4 for the HFs.

I was thinking of updating my UFs to not send CookedData into the HF layer and change the Splunktcp inputs to TCP inputs with disk based persistent queues. Here is the setting from the outputs.conf.spec file:

sendCookedData = [true|false]
* If true, events are cooked (have been processed by Splunk).
* If false, events are raw and untouched prior to sending.
* Set to false if you are sending to a third-party system.
* Defaults to true.

The goal is to gain the benefit of Persistent queuing to store data in an input queue to disk. This can help prevent data loss if the forwarder or indexer gets backed up. Does anyone have any thoughts on this item? There will be extra overhead on the HF layer due to uncooked data. I would overcome this by using a combination of Splunktcp and tcp inputs. TCP inputs would handle any data item that would need an extra level of data loss prevention.

I have not tried increasing the input queue memory size. This could be an alternate approach to disk based queues.

0 Karma

lguinn2
Legend

Personally, I don't think that you need this. If you want to do data loss prevention
1: Turn on the useAck option on every forwarder
2: Make sure that you are not overloading the middle forwarder tier
3: Size the overall forwarder queue appropriately (maxQueueSize) - this is especially necessary on the middle forwarder tier

Finally, you should not be using heavy forwarders as a middle tier unless you absolutely must. Universal forwarders are preferred as they are more efficient. I know that is counter-intuitive for most people, but it is true. Here is a very recent blog post that addresses this: Universal or Heavy, that is the question

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...