Hi All,
sorry if this is a stupid question.
When you configure a Intermediate Heavy Forwarder(Non-Indexing) receiver.
Can you add a persistent Que within this to avoid data loss for a period the upstream indexer is unavailable?
[splunktcp://9997]
persistentQueueSize=100MB <--- ???
disabled = 0
I've seen the docs and it's mostly around UF's and types of ephemeral connections, though it doesn't stipulate wither this can be done on the receiver. Though I imagine someone will tell me there's a better way to configure this.
Thanks in advance.
yes sir, you are on the right track
Filename: inputs.conf
Server: Intermediate forwarder
Input Stanza:
[some_input://<port_num>]
persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk
The persistent queue stores the data on the disk. By default, forwarders and indexers have an in-memory input queue of 500KB.
The steps to calculate the desired size of the persistent queue:
1. Predicted Traffic in GB /Number of servers(IF|HF) = Predicted Traffic on each server in GB
2. Predicted Traffic on each server / 24 (hours in a day) = data indexed/generated in 1 hour
3. If we consider the downtime of 1 hour, the value generated above( data indexed/generated in 1 hour ) should be the size of the persistent queue.
For example, suppose the daily data traffic is 400GB and there are 2 intermediate forwarders, each will have 200GB data.
If we want to have a persistent queue for the downtime of 1 hour, the size of the persistent queue should be 200/24 = 8.33 i.e ~8 GB.
hope it helps
Thanks for the detailed answer Adonio!
I'll give it ago and see how I get on.