Getting Data In

ERROR TcpOutputFd - Read error. Connection reset by peer : splunkforwarder

sanaa
New Member

Hi ,

I am pretty much new to Splunk. I want to forward audit.log of one of my Linux servers to view in Splunk Web. For this, I did the following steps:

1) Upgraded version of splunkforwarder to 6.4.2
2) Modified inputs.conf and outputs.conf
3) Restarted Splunk

But i am getting below logs in splunkd.log. Please let me know how to see these audit.logs in Splunk Web. Am I missing any steps?

08-23-2016 10:37:56.325 +0000 INFO  WatchedFile - Will begin reading at offset=5111808 for file='/opt/zenoss/log/audit.log'.
08-23-2016 10:37:56.626 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:26.227 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: NONE
08-23-2016 10:38:26.228 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:56.231 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:26.235 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:38.909 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 100 seconds.
08-23-2016 10:39:56.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:26.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:56.216 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:18.525 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 200 seconds.
08-23-2016 10:41:26.211 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:56.198 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:26.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:56.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:58.896 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 300 seconds.

Please help

0 Karma

vasanthmss
Motivator

Check your indexer version .. indexer should be high or equal version.. if not here are the few steps to troubleshoot,

  1. check your outputs.conf -

indexer ip - wrong ips / firewall issue

  1. telnet the indexer ip from forwarder and check the connection is valid or not? use the below
    telnet

    eg:

    telnet 10.99.0.1 9997

hope this will helps you.

thanks,
V

0 Karma

justynap_ldz
Explorer

@vasanthmss Do you have any other suggestions? 

We are working on Splunk 7.2.9.1. but encountered similar issue. 

ERROR TcpOutputFd - Read error. Connection reset by peer occured on one indexer. Splunkd stopped.

Then Splunk stopped on other 3 indexers that ended up with the following errors:
ERROR TcpOutputFd - Connection to host=xyzf failed and 
ERROR TcpOutputFd - Connect to host=xyzf refused. 

Also, in the same timeframe there was ClusterSlaveBucketHandler ERROR on one of the indexers.

Splunk version for all indexers is the same. I checked outputs.conf and run telnet between indexers. All fine.

Any hints will be much appreciated!

0 Karma