I want to see where is the path for persistent queues. I checked the following path on a heavy forwarder but I did not see any.
We get data from the universal forwarders where the splunk_windows_ta is installed and that sends data to a heavy forwarder and there to splunk.On which splunk instance(UF,HF,IDX) should I configure the persistent queue and what internal data tells when this queue gets filled
Thanks in advance
Easy one first - In your env you would want the PQ on the UF - It wont have any effect on Splunk forwarded data on an intermediate HF.
Secondly, the PQ path will not be the one specified above as that is for network inputs.
I have never fully understood why a PQ is needed for WinEventLogs - WinEventLogs are not at risk of being lost if blocked because they already exist on disk. If your UF is blocked, no WinEvent data is at risk of loss because it will just throttle back until there is capacity to start sending again (like a file monitor)
PQs make a lot of sense when data is inflight, or a backlog can have an impact upstream, but I'm not clear how it helps WinEventLogs
Do you have a specific issue which you need to use a PQ for on WinEventLogs?
Thinking a bit about this, iI suppose it offers some protection if you have been blocked for a long time, and the entire WinEvent Log buffer has filled up, and its about to start overwriting old events which Splunk has not yet committed.
Whilst this makes some sense, I cant really follow the logic to add an extra buffer of 100mb in Splunk, rather than add the 100mb to the Windows Event Log size. Chances are that if you have filled your log, and ingest has stopped, a 100mb PQ is unlikely to save the day, especially if the host gets restarted.
If log delivery is business critical, I'd make the winlogs bigger through GPO and turn on indexerAck.
Hello @nickhillscpl .I see in the splunk document which says you can use PQ for windows events logs
When can you use persistent queues?
Persistent queuing is available for certain types of inputs, but not all. Generally speaking, it is available for inputs of an ephemeral nature, such as network inputs, but not for inputs that have their own form of persistence, such as file monitoring.
Persistent queues are available for these input types:
Windows Event Log inputs
HTTP Event Collector tokens
The scenario happened with us was three of our 5 indexers were low on storage .We received messages about
1.minimum free disk space
2. TCPoutputproc has paused data flow and the forwarding to output group HF has been blocked for 3590 seconds.This will probably halt data flow towards indexing and other network outputs.It is probably not accepting data
3. Could not send data to output queue-queue filling
We removed the indexers on low storage and added new one .But in future we want to make sure to address this issues when occured .So what are the steps we can take
Making sure the indexers have disk space and where(UF,HF) should the queues increased and what location and what is the path for PQ and how do we set up it for windows event logs
From what you are describing I'm not sure that PQs would have been any real benefit, unless you are saying you now have gaps in your WinEventLogs?
In any case, even if you do have gaps in the WinEventLogs, I would still be tempted to increase the size (in windows) rather than in Splunk.
To answer your question, you need to define the PQ on your universal forwarder. (Not the HF)
in inputs.conf under your WinEventLog stanza:
[WinEventLog://<name>] persistentQueueSize = 100MB ... ...
interesting - this is not covered in the inputs.conf.spec though it is referred to in the link above. My guess is that this is not commonly used.
Thank you for your mail . Where can we increase the size (in windows) rather than in Splunk.Is it on UF or HF and what location
and what is this location for PQ $SPLUNK_HOME/var/run/splunk/[tcpin|udpin]/pq__
You would set it on your windows servers/workstations where you are worried about loosing logs.
You can also set it via a GPO.
$SPLUNK_HOME/var/run/splunk/tcpin/pq__1234 would be the path for a TCP input PQ on port 1234
$SPLUNK_HOME/var/run/splunk/udpin/pq__514 would be the path for a UDP input PQ on port 514