topic Re: Why do my forwarders have inconsistent connectivity to the indexers? in Getting Data In

Why do my forwarders have inconsistent connectivity to the indexers?

ddrillic — Thu, 04 Oct 2018 18:48:51 GMT

We have this case for multiple servers where we see constant errors such as -

10-04-2018 13:25:55.480 -0500 INFO  TcpOutputProc - Removing quarantine from idx=<indexer1>:9997
10-04-2018 13:25:55.480 -0500 INFO  TcpOutputProc - Removing quarantine from idx=<indexer2>:9997
10-04-2018 13:25:55.480 -0500 INFO  TcpOutputProc - Removing quarantine from idx=<indexer3>:9997
10-04-2018 13:25:55.480 -0500 INFO  TcpOutputProc - Removing quarantine from idx=<indexer4>:9997
10-04-2018 13:25:55.489 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
10-04-2018 13:25:55.491 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
10-04-2018 13:25:55.493 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
10-04-2018 13:25:55.501 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
10-04-2018 13:25:55.509 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer

telnet is fine -

telnet <indexer1> 9997
telnet <indexer2> 9997
telnet <indexer3> 9997
telnet <indexer4> 9997

We see lots of data for -

index=_internal host=<host name>

And nothing -

index=<customer index> host=<host name>

Re: Why do my forwarders have inconsistent connectivity to the indexers?

harsmarvania57 — Fri, 05 Oct 2018 11:42:43 GMT

Hi @ddrillic,

As you mentioned that connectivity looks good from UF to Indexer then I'll first check congestion on Indexer or Indexer Resource Usage (Might too busy - Resource constraint) to start troubleshooting.

Re: Why do my forwarders have inconsistent connectivity to the indexers?

ddrillic — Sun, 07 Oct 2018 19:14:31 GMT

Makes sense @harsmarvania57, but all other data sources look fine.

When searching for a specific path - index=_internal host=<host name> <path>

I see -

10-07-2018 14:04:24.766 -0500 INFO  Metrics - group=per_source_thruput, series="<path to log file>", kbps=0.05002527423023678, eps=0.419355447451456, kb=1.55078125, ev=13, avg_age=0.6153846153846154, max_age=3

What does it mean?

Re: Why do my forwarders have inconsistent connectivity to the indexers?

harsmarvania57 — Mon, 08 Oct 2018 09:06:16 GMT

This is metrics.log entry which indicates per_source_thruput from UF to Indexer for each and every source which you are monitoring. For detailed explanation of different fields in above log message please check https://docs.splunk.com/Documentation/Splunk/7.1.2/Troubleshooting/Aboutmetricslog#Thruput_messages

Re: Why do my forwarders have inconsistent connectivity to the indexers?

ddrillic — Fri, 12 Oct 2018 19:38:59 GMT

We do see the following error -

-- 10-12-2018 14:36:41.029 -0500 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group indexers has been blocked for 410303 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Re: Why do my forwarders have inconsistent connectivity to the indexers?

harsmarvania57 — Mon, 15 Oct 2018 09:45:53 GMT

This means that there might be possibility that different queues on Indexers are blocking for various reason, you need to check Monitoring Console for blocking queues status.