Getting Data In

Why am I getting these errors on my Splunk master node and Heavy forwarders for Splunkd?

eymanu
Explorer

Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

Audit event generator: Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly.

Root Cause: More than 70% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Last 50 related messages:
05-07-2018 13:30:34.005 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1580 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:24.089 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1570 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:14.070 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1560 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
05-07-2018 13:30:04.056 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group local_55153 has been blocked for 1550 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

TailReader-0
Root Cause: The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.
Last 50 related messages:
05-07-2018 13:04:20.241 -0400 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
05-07-2018 13:04:14.444 -0400 INFO TailReader - Starting batchreader0 thread
05-07-2018 13:04:14.444 -0400 INFO TailReader - Registering metrics callback for: batchreader0
05-07-2018 13:04:14.442 -0400 INFO TailReader - Starting tailreader0 thread
05-07-2018 13:04:14.442 -0400 INFO TailReader - Registering metrics callback for: tailreader0

jcrabb_splunk
Splunk Employee
Splunk Employee

Splunk on this host(s) is configured to forward its data to "local_55153" which is defined in outputs.conf:

Forwarding to output group local_55153 has been blocked for 1550 seconds.

If it is unable to connect to that output group, the data will begin to queue and once the queues are filled it will no longer process/ingest new data. You will want to review ouputs.conf on these hosts and determine if the settings are correct. If they are, then you will want to ensure that the instances that make up "local_55153" are reachable, are listening on the relevant port and are able to accept data.

Jacob
Sr. Technical Support Engineer
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...