WARN - TailReader - Could not send data to output queue (parsingQueue), retrying ???

[Jarohnimo]

Hello All,

"WARN - TailReader - Could not send data to output queue (parsingQueue), retrying"

^ I'm seeing this message in my _internal splunkd logs from my indexer

in my UF metrics log I see

Group=tailingprocessor, name=batchreader0, current_queue_size=254, max_queue_size=0, files_queued=-, new_files_queued=0

Looking at August 8th IIS Log for one log source = 7094030 events

doing a search on index time vs actual log time I see a difference of up to 2 hours for certain logs to come from our Universal forwarders. I've verified it's not an issues with the date/time stamp as I have my props.conf setting set for the sourcetype (iis). It's not an issue with the parsing queue on the indexer as there's no lag between the various stages. It seems to me that the forwarder is having some issues sending the data and I've done some research on this but not quite sure what the fix is.

I've read this article: https://docs.splunk.com/Documentation/Splunk/7.3.1/Troubleshooting/Troubleshootingeventsindexingdela...

however it states a different message would appear. For those who've seen this issue, is the fix simply to adjust the limits.conf on my universal forwarders? The only servers that are showing these issues are our Web Front ends that see heavy traffic.

I'd like to hear from someone who's seen this message before, and know what you've done to fix? Thanks

[DavidHourani]

Hi @Jarohnimo,

Best way to troubleshoot delays in indexing is the check the data pipeline health on the monitoring console.

Try to identify if any of your queues are full and then work from there to see what is causing the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Viewindexingperformance

Cheers,
David

[Jarohnimo]

You may have missed me stating this in my op

"It's not an issue with the parsing queue on the indexer as there's no lag between the various stages" as I've checked the monitoring console first. It's 0 across-the-board of each stage.

[DavidHourani]

What about the tail processor on the FWD? Have you checked if anything is blocked there ?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Troubleshoottheinputprocess#Troubleshoot_you...

You can consider adding parallel ingestion pipelines to allow more compute to be used on your UF :
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Pipelinesets

[Jarohnimo]

Thanks for that information about increasing the pipeline. I believe the issue is with the forwarders trying to process too much information. The indexer don't show queues between the various stages of the piplines.

I'm also curious now if our virtual team screwed us on our storage (gave us the slow stuff)

[DavidHourani]

ah shit... or shared LUN ... better get that confirmed first...