How to troubleshoot why universal forwarders are reporting "Could not send data to output queue (parsingQueue), retrying..."?

[prakash007]

I'm getting this message below on Universal Forwarders' splunkd.log...

INFO  BatchReader - Could not send data to output queue (parsingQueue), retrying...
INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...

I did follow this step below...

1. grep "*blocked=true*" /opt/app/splunkforwarder/var/log/splunk/metrics.log* I don't see any blocked queues
2. I did add limits.conf in /opt/apps/splunkforwarder/etc/system/local [thruput] maxKBps = 0

Still I see the message:

Could not send data to output queue (parsingQueue), retrying...

What are the next options I need to look to resolve this..??

[muebel]

Hi mcnamara, The next options will be to verify that the forwarder has connectivity to the upstream tcpout host. This can be done by using telnet or openssl commands

openssl s_client -connect <upstreamhost>:<port>

Additionally, look at other universal forwarder installations and determine if they are able to connect. If they can, then that means that you have a problem with the one particular host in question. Otherwise there is an issue with the overall outputs.conf configuration, or a networking issue (simply no route to upstream splunk instance).

Please let me know if this helps!

[martin_mueller]

The message itself says "outputqueue on forwarder is full", but that's usually just a symptom. The cause usually is no connectivity to the indexing tier, or full queues on the indexing tier, or some other indexing blockage.

[prakash007]

Hi muebel, I did try your command and it says connected

$ openssl s_client -connect apwebsvr:9997
CONNECTED(00000003)
3648:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Here's our data flow... UF------->HF-------->Splunkcloud, i did telnet and openssl from UF to HF which is connecting.

I don't see this message (Could not send data to output queue (parsingQueue), retrying..) when i restart the splunk instance on UF, but it's been happening every now and then.

Based on the message in the log, is parsingQueue gets filled up on UF or HF or Indexer...? just trying to understand to get a permanent solution. Thanks..!!

[somesoni2]

Is your forwarder able to connect to Indexer? Check the firewall rules etc..

[prakash007]

Yes it is connecting, i did $telnet servername port#