Getting Data In

WARN - TailReader - Could not send data to output queue (parsingQueue), retrying ???

Jarohnimo
Builder

Hello All,

"WARN - TailReader - Could not send data to output queue (parsingQueue), retrying"

^ I'm seeing this message in my _internal splunkd logs from my indexer

in my UF metrics log I see

Group=tailingprocessor, name=batchreader0, current_queue_size=254, max_queue_size=0, files_queued=-, new_files_queued=0

Looking at August 8th IIS Log for one log source = 7094030 events

doing a search on index time vs actual log time I see a difference of up to 2 hours for certain logs to come from our Universal forwarders. I've verified it's not an issues with the date/time stamp as I have my props.conf setting set for the sourcetype (iis). It's not an issue with the parsing queue on the indexer as there's no lag between the various stages. It seems to me that the forwarder is having some issues sending the data and I've done some research on this but not quite sure what the fix is.

I've read this article: https://docs.splunk.com/Documentation/Splunk/7.3.1/Troubleshooting/Troubleshootingeventsindexingdela...

however it states a different message would appear. For those who've seen this issue, is the fix simply to adjust the limits.conf on my universal forwarders? The only servers that are showing these issues are our Web Front ends that see heavy traffic.

I'd like to hear from someone who's seen this message before, and know what you've done to fix? Thanks

0 Karma

DavidHourani
Super Champion

Hi @Jarohnimo,

Best way to troubleshoot delays in indexing is the check the data pipeline health on the monitoring console.

Try to identify if any of your queues are full and then work from there to see what is causing the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Viewindexingperformance

Cheers,
David

0 Karma

Jarohnimo
Builder

You may have missed me stating this in my op

"It's not an issue with the parsing queue on the indexer as there's no lag between the various stages" as I've checked the monitoring console first. It's 0 across-the-board of each stage.

0 Karma

DavidHourani
Super Champion

What about the tail processor on the FWD? Have you checked if anything is blocked there ?
https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Troubleshoottheinputprocess#Troubleshoot_you...

You can consider adding parallel ingestion pipelines to allow more compute to be used on your UF :
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Pipelinesets

0 Karma

Jarohnimo
Builder

Thanks for that information about increasing the pipeline. I believe the issue is with the forwarders trying to process too much information. The indexer don't show queues between the various stages of the piplines.

I'm also curious now if our virtual team screwed us on our storage (gave us the slow stuff)

0 Karma

DavidHourani
Super Champion

ah shit... or shared LUN ... better get that confirmed first...

0 Karma
Get Updates on the Splunk Community!

Unlock New Opportunities with Splunk Education: Explore Our Latest Courses!

At Splunk Education, we’re dedicated to providing top-tier learning experiences that cater to every skill ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...