Forwarder: TcpOutputProc ... Connection closed by server

[dbablinyuk]

The log

07-22-2011 15:04:38.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:38.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.693 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
...

Not sure how to get over it.

cat /opt/splunkforwarder/etc/system/default/outputs.conf 

[tcpout]
server = 172.16.40.116:9997
disabled = false
compressed = true

cat /opt/splunkforwarder/etc/system/default/inputs.conf
...
[monitor:///var/log/messages]
disabled = false
index = _internal
sourcetype = linux_messages_syslog

Server does not receive anything.

Really appreciate help on this

Thanks!

Dimitry

[dbablinyuk]

Thank you very much Sean!

It worked!

The issue was that the encryption was turned on for the forwarder but not for the receiver.

Thank you again

Dimitry

[boopaljothi]

is the inputs.conf file in etc\system\local that you have verified in forwarder? what would need to be changed

[sdwilkerson]

Dimitry,

I see a problem in your config and there could be several things preventing access.

First, the problem. I noticed in your included inputs.conf that you force events discovered in "/var/log/messages" to go to index=_internal. The _internal index is very special and generally reserved for Splunk-internal logs (hence the name). If you omitted this line entirely, a default instance of Splunk would automatically place new events into the default index which is index=main. You might want to omit this line or create a new index and use that since _internal is definitely not the right place for your data.

On to your reported issue here are some bullets to consider or help you troubleshoot.

• Is the "server" configured to listen on port 9997?
• If not configured to listen, then run this command on the server/indexer: $SPLUNK_HOME/bin/splunk enable listen
• If you did configure it to listen, then attempt manually to connect from a sender (forwarder) to the receiver (indexer) by doing something like this on the sender: telnet 172.16.40.116 9997
• This telnet is to see if you have TCP access and connectivity from the sender to the receiver on port tcp/9997. If you see your telnet session "connect" and go to a blank screen then this test was successful and you are having some other problem. However, if you see your telnet session hang at something like "cannot connect to host" or "connection to host refused" or something like that, then you might have an issue with a firewall, router, or access control either on or between the sender and the receiver.
• On the receiver (indexer), does the $SPLUNK_HOME/var/log/splunk/splunkd.log show the connection attempts from the sender(s)?
• Do you have encryption or compression configured on the side of the receiver but not on the sender? Note: if you did, this would be configured on the "input.conf" on the receiver.

Sean

[boopaljothi]

i have the same problem and in my case it is set up fine.

enabled firewall on the port 9997
telnet is working from forwarder to receiver
there is no encryption settings as per the inputs.conf file. i just have the default data in the inputs.conf file when i installed it.
also the forwarders has the data as below
[default]
host = xxxx

[WinEventLog://Application]
disabled = 0
index = xxxx
sourcetype = security

receiver splunkd log is not being updated with senders information

could you please help me in fixing this.