Deployment Architecture

Splunkforwarder 6.1.4 blocks and unable to self-recover "TailingProcessor - Could not send data to output queue (parsingQueue), retrying..."

the_wolverine
Champion

Seen in splunk.log repeatedly (nothing else)

TailingProcessor - Could not send data to output queue (parsingQueue), retrying...

Our forwarders seem to get blocked occasionally and are unable to recover. We've found them in this state for days sometimes, and due to block, we don't get the internal logs in Splunk to detect this condition.

Files monitored:
maybe 50-100 files, 10 active files, rolled files ~100MB each at rotation.

A restart of the splunkforwarder resolves the issue.

woodcock
Esteemed Legend

If what @ddrillic (@lguinn) said is the problem; here is another way out:

https://answers.splunk.com/answers/309910/how-to-monitor-a-folder-for-newest-files-only-file.html

0 Karma

ddrillic
Ultra Champion

The question is whether they are being blocked at the forwarder level or at the indexer level.

Cheerful discussion at Could not send data to output queue (parsingQueue)

@lguinn explained and said -

alt text

0 Karma

the_wolverine
Champion

There are maybe 50-100 (100MB) files so this is not the issue. Also, it is forwarder-specific .. as in a handful of forwarders get blocked and never recover on their own.

Occasional blocking at the indexer is normal and recovers. But am seeing ceilings being hit, but indexer recovers .. but the forwarder does not for many days before I detect it. Restarting the splunkforwarder resolves it.

0 Karma

ben_leung
Builder

Is it just parsing queue thats blocked?

0 Karma

woodcock
Esteemed Legend

I don't know what is causing it but you can turn on an alert inside the (Deployment) Monitoring Console to alert you to Missing Forwarders. If you can cannot find it, run the Health Checks and look for bread crumbs there.

0 Karma

woodcock
Esteemed Legend

Also, upgrade to the latest forwarder version of Splunk; I find that forwarders often run VERY far behind in versions, which is not good.

0 Karma

the_wolverine
Champion

Yeah, we're planning on upgrading soon as assuming this is an undocumented bug at this point.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...