Getting Data In

WARN - TailReader - Could not send data to output queue (parsingQueue), retrying ???

Jarohnimo
Builder

Hello All,

"WARN - TailReader - Could not send data to output queue (parsingQueue), retrying"

^ I'm seeing this message in my _internal splunkd logs from my indexer

in my UF metrics log I see

Group=tailingprocessor, name=batchreader0, current_queue_size=254, max_queue_size=0, files_queued=-, new_files_queued=0

Looking at August 8th IIS Log for one log source = 7094030 events

doing a search on index time vs actual log time I see a difference of up to 2 hours for certain logs to come from our Universal forwarders. I've verified it's not an issues with the date/time stamp as I have my props.conf setting set for the sourcetype (iis). It's not an issue with the parsing queue on the indexer as there's no lag between the various stages. It seems to me that the forwarder is having some issues sending the data and I've done some research on this but not quite sure what the fix is.

I've read this article: https://docs.splunk.com/Documentation/Splunk/7.3.1/Troubleshooting/Troubleshootingeventsindexingdela...

however it states a different message would appear. For those who've seen this issue, is the fix simply to adjust the limits.conf on my universal forwarders? The only servers that are showing these issues are our Web Front ends that see heavy traffic.

I'd like to hear from someone who's seen this message before, and know what you've done to fix? Thanks

0 Karma

DavidHourani
Super Champion

Hi @Jarohnimo,

Best way to troubleshoot delays in indexing is the check the data pipeline health on the monitoring console.

Try to identify if any of your queues are full and then work from there to see what is causing the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Viewindexingperformance

Cheers,
David

0 Karma

Jarohnimo
Builder

You may have missed me stating this in my op

"It's not an issue with the parsing queue on the indexer as there's no lag between the various stages" as I've checked the monitoring console first. It's 0 across-the-board of each stage.

0 Karma

DavidHourani
Super Champion

What about the tail processor on the FWD? Have you checked if anything is blocked there ?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Troubleshoottheinputprocess#Troubleshoot_you...

You can consider adding parallel ingestion pipelines to allow more compute to be used on your UF :
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Pipelinesets

0 Karma

Jarohnimo
Builder

Thanks for that information about increasing the pipeline. I believe the issue is with the forwarders trying to process too much information. The indexer don't show queues between the various stages of the piplines.

I'm also curious now if our virtual team screwed us on our storage (gave us the slow stuff)

0 Karma

DavidHourani
Super Champion

ah shit... or shared LUN ... better get that confirmed first...

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...