Getting Data In

How to find the path for persistent queues

vrmandadi
Builder

I want to see where is the path for persistent queues. I checked the following path on a heavy forwarder but I did not see any.

$SPLUNK_HOME/var/run/splunk/[tcpin|udpin]/pq__

We get data from the universal forwarders where the splunk_windows_ta is installed and that sends data to a heavy forwarder and there to splunk.On which splunk instance(UF,HF,IDX) should I configure the persistent queue and what internal data tells when this queue gets filled

Thanks in advance

0 Karma

nickhills
Ultra Champion

Easy one first - In your env you would want the PQ on the UF - It wont have any effect on Splunk forwarded data on an intermediate HF.

Secondly, the PQ path will not be the one specified above as that is for network inputs.

I have never fully understood why a PQ is needed for WinEventLogs - WinEventLogs are not at risk of being lost if blocked because they already exist on disk. If your UF is blocked, no WinEvent data is at risk of loss because it will just throttle back until there is capacity to start sending again (like a file monitor)

PQs make a lot of sense when data is inflight, or a backlog can have an impact upstream, but I'm not clear how it helps WinEventLogs

Do you have a specific issue which you need to use a PQ for on WinEventLogs?

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

Thinking a bit about this, iI suppose it offers some protection if you have been blocked for a long time, and the entire WinEvent Log buffer has filled up, and its about to start overwriting old events which Splunk has not yet committed.

Whilst this makes some sense, I cant really follow the logic to add an extra buffer of 100mb in Splunk, rather than add the 100mb to the Windows Event Log size. Chances are that if you have filled your log, and ingest has stopped, a 100mb PQ is unlikely to save the day, especially if the host gets restarted.

If log delivery is business critical, I'd make the winlogs bigger through GPO and turn on indexerAck.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

Hello @nickhillscpl .I see in the splunk document which says you can use PQ for windows events logs
When can you use persistent queues?
Persistent queuing is available for certain types of inputs, but not all. Generally speaking, it is available for inputs of an ephemeral nature, such as network inputs, but not for inputs that have their own form of persistence, such as file monitoring.

Persistent queues are available for these input types:

TCP
UDP
FIFO
Scripted inputs
Windows Event Log inputs
HTTP Event Collector tokens

https://docs.splunk.com/Documentation/Splunk/8.0.1/Data/Usepersistentqueues

The scenario happened with us was three of our 5 indexers were low on storage .We received messages about
1.minimum free disk space
2. TCPoutputproc has paused data flow and the forwarding to output group HF has been blocked for 3590 seconds.This will probably halt data flow towards indexing and other network outputs.It is probably not accepting data
3. Could not send data to output queue-queue filling

We removed the indexers on low storage and added new one .But in future we want to make sure to address this issues when occured .So what are the steps we can take

Making sure the indexers have disk space and where(UF,HF) should the queues increased and what location and what is the path for PQ and how do we set up it for windows event logs

0 Karma

nickhills
Ultra Champion

From what you are describing I'm not sure that PQs would have been any real benefit, unless you are saying you now have gaps in your WinEventLogs?

In any case, even if you do have gaps in the WinEventLogs, I would still be tempted to increase the size (in windows) rather than in Splunk.

To answer your question, you need to define the PQ on your universal forwarder. (Not the HF)
in inputs.conf under your WinEventLog stanza:

[WinEventLog://<name>]
persistentQueueSize = 100MB
...
...

interesting - this is not covered in the inputs.conf.spec though it is referred to in the link above. My guess is that this is not commonly used.

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

Thank you for your mail . Where can we increase the size (in windows) rather than in Splunk.Is it on UF or HF and what location

and what is this location for PQ $SPLUNK_HOME/var/run/splunk/[tcpin|udpin]/pq__

0 Karma

nickhills
Ultra Champion

You would set it on your windows servers/workstations where you are worried about loosing logs.

https://docs.microsoft.com/en-gb/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/dd...

You can also set it via a GPO.

Your question:
$SPLUNK_HOME/var/run/splunk/tcpin/pq__1234 would be the path for a TCP input PQ on port 1234
$SPLUNK_HOME/var/run/splunk/udpin/pq__514 would be the path for a UDP input PQ on port 514

If my comment helps, please give it a thumbs up!
0 Karma

vrmandadi
Builder

The link you sent says page not found

0 Karma

nickhills
Ultra Champion

https://support.microsoft.com/kb/957662

If my comment helps, please give it a thumbs up!
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!