Deployment Architecture

Splunkforwarder 6.1.4 blocks and unable to self-recover "TailingProcessor - Could not send data to output queue (parsingQueue), retrying..."

the_wolverine
Champion

Seen in splunk.log repeatedly (nothing else)

TailingProcessor - Could not send data to output queue (parsingQueue), retrying...

Our forwarders seem to get blocked occasionally and are unable to recover. We've found them in this state for days sometimes, and due to block, we don't get the internal logs in Splunk to detect this condition.

Files monitored:
maybe 50-100 files, 10 active files, rolled files ~100MB each at rotation.

A restart of the splunkforwarder resolves the issue.

woodcock
Esteemed Legend

If what @ddrillic (@lguinn) said is the problem; here is another way out:

https://answers.splunk.com/answers/309910/how-to-monitor-a-folder-for-newest-files-only-file.html

0 Karma

ddrillic
Ultra Champion

The question is whether they are being blocked at the forwarder level or at the indexer level.

Cheerful discussion at Could not send data to output queue (parsingQueue)

@lguinn explained and said -

alt text

0 Karma

the_wolverine
Champion

There are maybe 50-100 (100MB) files so this is not the issue. Also, it is forwarder-specific .. as in a handful of forwarders get blocked and never recover on their own.

Occasional blocking at the indexer is normal and recovers. But am seeing ceilings being hit, but indexer recovers .. but the forwarder does not for many days before I detect it. Restarting the splunkforwarder resolves it.

0 Karma

ben_leung
Builder

Is it just parsing queue thats blocked?

0 Karma

woodcock
Esteemed Legend

I don't know what is causing it but you can turn on an alert inside the (Deployment) Monitoring Console to alert you to Missing Forwarders. If you can cannot find it, run the Health Checks and look for bread crumbs there.

0 Karma

woodcock
Esteemed Legend

Also, upgrade to the latest forwarder version of Splunk; I find that forwarders often run VERY far behind in versions, which is not good.

0 Karma

the_wolverine
Champion

Yeah, we're planning on upgrading soon as assuming this is an undocumented bug at this point.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...