Forwarder not working

[Lazarix]

Hi all,
Not sure what i'm doing wrong exactly.

I have a a linux server set up with the universal forwarder and I have it set to forward to two splunk servers. On One server (windows) it works fine and is receiving all the input. On the second server (linux) it's not working.

The forwarder's splunkd.log has the following information:


06-30-2012 14:34:42.083 +0100 WARN TcpOutputProc - Write to 10.240.24.220:9996 failed. Write failed. Connection reset by peer
06-30-2012 14:34:42.088 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:35:50.155 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Connection closed by server.
06-30-2012 14:35:50.161 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:35:50.166 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Read error. Connection reset by peer
06-30-2012 14:35:50.166 +0100 WARN TcpOutputProc - Applying quarantine to idx=10.240.24.220:9996 numberOfFailures=2
06-30-2012 14:36:11.001 +0100 INFO TcpOutputProc - Removing quarantine from idx=10.240.24.220:9996
06-30-2012 14:36:11.007 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:36:11.012 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Read error. Connection reset by peer
06-30-2012 14:36:11.012 +0100 WARN TcpOutputProc - Applying quarantine to idx=10.240.24.220:9996 numberOfFailures=3
06-30-2012 14:36:41.002 +0100 INFO TcpOutputProc - Removing quarantine from idx=10.240.24.220:9996
06-30-2012 14:36:41.007 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:36:41.013 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Read error. Connection reset by peer
06-30-2012 14:36:41.014 +0100 WARN TcpOutputProc - Applying quarantine to idx=10.240.24.220:9996 numberOfFailures=4
06-30-2012 14:37:41.004 +0100 INFO TcpOutputProc - Removing quarantine from idx=10.240.24.220:9996
06-30-2012 14:37:41.009 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:37:41.016 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Read error. Connection reset by peer
06-30-2012 14:37:41.016 +0100 WARN TcpOutputProc - Applying quarantine to idx=10.240.24.220:9996 numberOfFailures=5
06-30-2012 14:37:46.602 +0100 INFO TcpOutputProc - Queue for group Splunk has begun dropping events
06-30-2012 14:39:11.005 +0100 INFO TcpOutputProc - Removing quarantine from idx=10.240.24.220:9996
06-30-2012 14:39:11.011 +0100 INFO TcpOutputProc - Connected to idx=10.240.24.220:9996
06-30-2012 14:39:11.017 +0100 INFO TcpOutputProc - Connection to 10.240.24.220:9996 closed. Connection closed by server.
06-30-2012 14:39:11.017 +0100 WARN TcpOutputProc - Applying quarantine to idx=10.240.24.220:9996 numberOfFailures=6


I confirm that i'm able to telnet to port 9996 from the forwarding server and I get no error.

[Lazarix · Accepted solution]

So, I think i'm suffering from the same problem as this:

http://splunk-base.splunk.com/answers/11172/intermediate-forwarder-problems-timestamping-transforms

and my "solution" was to set "sendCookedData" = true. Only then would it receive data into the Linux splunk server.

Is there a way to fix this?

[Lazarix]

What's even more frustrating is that on my spunk server (linux) in splunkd.conf I can see some messages for events which are coming from the forwarder but it's almost like it sends just one message and then craps out.

06-30-2012 15:40:03.381 +0100 ERROR TcpInputProc - Received unrecognized signature INFO main com.batchengine.BatchEngineManager - [Sat Jun 30 15:38:58 BST 2012] Details object is updated with active values for jobs/ret/app/cli/usr
DEBUG main com.batchengine.BatchEngineManager - In BEM Section Retailer Details max object are : 60:42:26:13:6:3
DEBUG main com.batchengine.BatchEngineManager - In BEM Section Retailer Details active object are : 13:1:0:0:0:0
DEBUG! from src=205.217.18.196:35785
06-30-2012 15:40:33.385 +0100 ERROR TcpInputProc - Received unrecognized signature
INFO pool-1-thread-2 com.batchengine.DatabaseManager - ProcessDMLStatement : 1 : UPDATE ASPAUDITHEADER SET LASTENTRY = 'Common header grouping cleared', TOTALINSERT = TOTALINSERT+0, TOTALUPDATE = TOTALUPDATE+0, TOTALDELETE = TOTALDELETE+0 WHERE JOBID = 1079030 AND LOGID = 20120630152639;
INFO pool-1-thread-2 com.batchengine.BatchEngineManager - [Sat Jun 30 15:38:58 BST 2012] S! from src=205.217.18.196:35820
06-30-2012 15:41:03.387 +0100 ERROR TcpInputProc - Received unrecognized signature
06-30-2012 15:38:58.125 +0100 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
06-30-2012 15:38:58.125 +0100 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
06-30-2012 15:38:58.125 +0100 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
06-30-2012 15:38:58.125 +0100 INFO ShutdownHandler - Component 'TcpOutputProce! from src=205.217.18.196:35864
06-30-2012 15:42:03.388 +0100 ERROR TcpInputProc - Received unrecognized signature 06-30-2012 15:38:58.125 +0100 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
06-30-2012 15:38:58.125 +0100 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
06-30-2012 15:38:58.125 +0100 INFO TcpOutputProc - begin to shut down auto load balanced connection strategy
06-30-2012 15:38:58.125 +0100 INFO ShutdownHandler - Component 'TcpOutputProces! from src=205.217.18.196:35969

[Lazarix]

My outputs.conf is like so:


[tcpout]
defaultGroup = testSplunk,Splunk

[tcpout:testSplunk]
server=server-01:9997
sendCookedData=false

[tcpout:Splunk]
server=10.240.24.220:9996
sendCookedData=false

The non-working server is listed as "Splunk"