Getting Data In

Why am I getting these errors on my Splunk master node and Heavy forwarders for Splunkd?

eymanu
Explorer

Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

Root Cause: More than 70% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Last 50 related messages:
05-07-2018 13:30:34.005 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1580 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:24.089 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1570 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:14.070 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1560 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:04.056 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1550 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

TailReader-0
Root Cause: The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.
Last 50 related messages:
05-07-2018 13:04:20.241 -0400 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
05-07-2018 13:04:14.444 -0400 INFO TailReader - Starting batchreader0 thread
05-07-2018 13:04:14.444 -0400 INFO TailReader - Registering metrics callback for: batchreader0
05-07-2018 13:04:14.442 -0400 INFO TailReader - Starting tailreader0 thread
05-07-2018 13:04:14.442 -0400 INFO TailReader - Registering metrics callback for: tailreader0

jcrabb_splunk
Splunk Employee
Splunk Employee

Splunk on this host(s) is configured to forward its data to "local_55153" which is defined in outputs.conf:

Forwarding to output group local_55153 has been blocked for 1550 seconds.

If it is unable to connect to that output group, the data will begin to queue and once the queues are filled it will no longer process/ingest new data. You will want to review ouputs.conf on these hosts and determine if the settings are correct. If they are, then you will want to ensure that the instances that make up "local_55153" are reachable, are listening on the relevant port and are able to accept data.

Jacob
Sr. Technical Support Engineer
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...