ERROR TcpOutputFd - Read error. Connection reset by peer : splunkforwarder

[sanaa]

Hi ,

I am pretty much new to Splunk. I want to forward audit.log of one of my Linux servers to view in Splunk Web. For this, I did the following steps:

1) Upgraded version of splunkforwarder to 6.4.2
2) Modified inputs.conf and outputs.conf
3) Restarted Splunk

But i am getting below logs in splunkd.log. Please let me know how to see these audit.logs in Splunk Web. Am I missing any steps?

08-23-2016 10:37:56.325 +0000 INFO  WatchedFile - Will begin reading at offset=5111808 for file='/opt/zenoss/log/audit.log'.
08-23-2016 10:37:56.626 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:26.227 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: NONE
08-23-2016 10:38:26.228 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:56.231 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:26.235 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:38.909 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 100 seconds.
08-23-2016 10:39:56.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:26.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:56.216 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:18.525 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 200 seconds.
08-23-2016 10:41:26.211 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:56.198 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:26.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:56.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:58.896 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 300 seconds.

Please help

[woodcock]

Here are some things that hopefully you can change/disable that can get in the way:
FIPS
selinux
firewall (firewalld)
missing route
dns

[sraymondg]

I tried to disable FIPS on Splunk forwarder as it looks like FIPS is disabled on Splunk cloud or indexer also any forwarder with FIPS turned on will fail to be allowed to connect.

On the mis-configured forwarders disable FIPS and reboot.

Check FIPS is disabled with the next command:

cat /proc/sys/crypto/fips_enabled 0

https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/SecuringSplunkEnterprisewithFIPS

The Federal Information Processing Standard (FIPS) uses government-certified versions of some algorithms to meet regulatory guidelines.
It should not be considered a security enhancement by itself, and might potentially reduce performance on your system.
Enable FIPS if it is a regulatory requirement for your environment.
Splunk Enterprise and the Universal Forwarder use an embedded FIPS 140-2-validated cryptographic module.
Thus you need FIPS enabled and running on both the Forwarder side and the Indexer side

[vasanthmss]

Check your indexer version .. indexer should be high or equal version.. if not here are the few steps to troubleshoot,

1. check your outputs.conf -

indexer ip - wrong ips / firewall issue

1. telnet the indexer ip from forwarder and check the connection is valid or not? use the below
   telnet
   
   eg:

   telnet 10.99.0.1 9997

hope this will helps you.

thanks,
V

[justynap_ldz]

@vasanthmss Do you have any other suggestions? 

We are working on Splunk 7.2.9.1. but encountered similar issue. 

ERROR TcpOutputFd - Read error. Connection reset by peer occured on one indexer. Splunkd stopped.

Then Splunk stopped on other 3 indexers that ended up with the following errors:
ERROR TcpOutputFd - Connection to host=xyzf failed and 
ERROR TcpOutputFd - Connect to host=xyzf refused. 

Also, in the same timeframe there was ClusterSlaveBucketHandler ERROR on one of the indexers.

Splunk version for all indexers is the same. I checked outputs.conf and run telnet between indexers. All fine.

Any hints will be much appreciated!

[Sanjayr1081]

Hi,

I am having the same issue.

Logs are not going to index from forwarder and I am getting same error.

Did you got any solution for this? @justynap_ldz