Can you configure the Receiver stanza to have a Persistent Queue?

[kuroai]

Hi All,

sorry if this is a stupid question.

When you configure a Intermediate Heavy Forwarder(Non-Indexing) receiver.
Can you add a persistent Que within this to avoid data loss for a period the upstream indexer is unavailable?

[splunktcp://9997]
persistentQueueSize=100MB <--- ???
disabled = 0

I've seen the docs and it's mostly around UF's and types of ephemeral connections, though it doesn't stipulate wither this can be done on the receiver. Though I imagine someone will tell me there's a better way to configure this.

Thanks in advance.

[adonio]

yes sir, you are on the right track

Filename: inputs.conf
Server: Intermediate forwarder
Input Stanza:
[some_input://<port_num>]
persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk

The persistent queue stores the data on the disk. By default, forwarders and indexers have an in-memory input queue of 500KB.

The steps to calculate the desired size of the persistent queue:
1. Predicted Traffic in GB /Number of servers(IF|HF) = Predicted Traffic on each server in GB
2. Predicted Traffic on each server / 24 (hours in a day) = data indexed/generated in 1 hour
3. If we consider the downtime of 1 hour, the value generated above( data indexed/generated in 1 hour ) should be the size of the persistent queue.

For example, suppose the daily data traffic is 400GB and there are 2 intermediate forwarders, each will have 200GB data.
If we want to have a persistent queue for the downtime of 1 hour, the size of the persistent queue should be 200/24 = 8.33 i.e ~8 GB.

hope it helps

[rajashaey]

Can I configure something like this? 

[default]
persistentQueueSize=100MB

so that it will applied for all the inputs?

[PickleRick]

1. This is a very old thread. Starting a new one would give you more visibility.

2. Well, not every type of input supports this parameter so I'm not sure if specifying it here is syntactically correct. Try and see (with btool check)

[kuroai]

Thanks for the detailed answer Adonio!

I'll give it ago and see how I get on.