Getting Data In

Can you configure the Receiver stanza to have a Persistent Queue?

kuroai
New Member

Hi All,

sorry if this is a stupid question.

When you configure a Intermediate Heavy Forwarder(Non-Indexing) receiver.
Can you add a persistent Que within this to avoid data loss for a period the upstream indexer is unavailable?

[splunktcp://9997]
persistentQueueSize=100MB <--- ???
disabled = 0

I've seen the docs and it's mostly around UF's and types of ephemeral connections, though it doesn't stipulate wither this can be done on the receiver. Though I imagine someone will tell me there's a better way to configure this.

Thanks in advance.

0 Karma

adonio
Ultra Champion

yes sir, you are on the right track

Filename: inputs.conf
Server: Intermediate forwarder
Input Stanza:
[some_input://<port_num>]
persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk

The persistent queue stores the data on the disk. By default, forwarders and indexers have an in-memory input queue of 500KB.

The steps to calculate the desired size of the persistent queue:
1. Predicted Traffic in GB /Number of servers(IF|HF) = Predicted Traffic on each server in GB
2. Predicted Traffic on each server / 24 (hours in a day) = data indexed/generated in 1 hour
3. If we consider the downtime of 1 hour, the value generated above( data indexed/generated in 1 hour ) should be the size of the persistent queue.

For example, suppose the daily data traffic is 400GB and there are 2 intermediate forwarders, each will have 200GB data.
If we want to have a persistent queue for the downtime of 1 hour, the size of the persistent queue should be 200/24 = 8.33 i.e ~8 GB.

hope it helps

0 Karma

kuroai
New Member

Thanks for the detailed answer Adonio!

I'll give it ago and see how I get on.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...