https://docs.splunk.com/Documentation/Splunk/9.4.0/ReleaseNotes/MeetSplunk#What.27s_New_in_9.4
Why New Splunk TcpOutput Persistent Queue?
- Scheduled no connectivity for extended period but need to resume data transmission once connection is back up. Assuming there is enough storage, tcpout output queue can persist all events to disk instead of buying expensive third party subscription(unsupported) to persist data to S3.
- If there are two tcpout output destinations and one is down for extended period. Down destination has large enough PQ to persist data, then second destination is not blocked. Second destination will block only once PQ of down destination is full.
- Don't have to pay for third party PQ & S3 puts.
- Third party/ external S3 queue introduces permanent additional latency( due to detour to external S3 queue).
- Third party/ external S3 queuing requires batching events, which adds additional latency in order to reduce S3 puts cost.
- Unwanted additional network bandwidth usage incurred due to uploading all data to S3 and then downloading .
- Third party imposes upload payload size limits.
- Monitored corporate laptops are off network, not connected to internet or not connected to VPN for extended period of time. Later laptops might get switched off but events should be persisted and forwarded as and when laptop connects to network.
- Sensitive data should stay/persisted within network.
- On demand persistent queuing on forwarding tier when Indexer Clustering is down.
- On demand persistent queuing on forwarding tier when Indexer Clustering indexing is slow due to high system load.
On demand persistent queuing on forwarding tier when Indexer Clustering is in rolling restart.
How to enable?
Just set persistentQueueSize as per outputs.conf
[tcpout:splunk-group1]
persistentQueueSize=1TB
[tcpout:splunk-group2]
persistentQueueSize=2TB
Note: Sizing guide coming soon.
