Getting Data In

Forwarder: TcpOutputProc ... Connection closed by server

dbablinyuk
Engager
The log

07-22-2011 15:04:38.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:38.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.693 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connected to idx=172.16.40.116:9997
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
07-22-2011 15:04:41.694 +1000 INFO  TcpOutputProc - Connection to 172.16.40.116:9997 closed. Connection closed by server.
...

Not sure how to get over it.

cat /opt/splunkforwarder/etc/system/default/outputs.conf 

[tcpout]
server = 172.16.40.116:9997
disabled = false
compressed = true

cat /opt/splunkforwarder/etc/system/default/inputs.conf
...
[monitor:///var/log/messages]
disabled = false
index = _internal
sourcetype = linux_messages_syslog

Server does not receive anything.

Really appreciate help on this

Thanks!

Dimitry
Tags (2)

dbablinyuk
Engager

Thank you very much Sean!

It worked!

The issue was that the encryption was turned on for the forwarder but not for the receiver.

Thank you again

Dimitry

0 Karma

boopaljothi
Explorer

is the inputs.conf file in etc\system\local that you have verified in forwarder? what would need to be changed

0 Karma

sdwilkerson
Contributor

Dimitry,

I see a problem in your config and there could be several things preventing access.

First, the problem. I noticed in your included inputs.conf that you force events discovered in "/var/log/messages" to go to index=_internal. The _internal index is very special and generally reserved for Splunk-internal logs (hence the name). If you omitted this line entirely, a default instance of Splunk would automatically place new events into the default index which is index=main. You might want to omit this line or create a new index and use that since _internal is definitely not the right place for your data.

On to your reported issue here are some bullets to consider or help you troubleshoot.

  • Is the "server" configured to listen on port 9997?
  • If not configured to listen, then run this command on the server/indexer: $SPLUNK_HOME/bin/splunk enable listen
  • If you did configure it to listen, then attempt manually to connect from a sender (forwarder) to the receiver (indexer) by doing something like this on the sender: telnet 172.16.40.116 9997
  • This telnet is to see if you have TCP access and connectivity from the sender to the receiver on port tcp/9997. If you see your telnet session "connect" and go to a blank screen then this test was successful and you are having some other problem. However, if you see your telnet session hang at something like "cannot connect to host" or "connection to host refused" or something like that, then you might have an issue with a firewall, router, or access control either on or between the sender and the receiver.
  • On the receiver (indexer), does the $SPLUNK_HOME/var/log/splunk/splunkd.log show the connection attempts from the sender(s)?
  • Do you have encryption or compression configured on the side of the receiver but not on the sender? Note: if you did, this would be configured on the "input.conf" on the receiver.

Sean

boopaljothi
Explorer

i have the same problem and in my case it is set up fine.

enabled firewall on the port 9997
telnet is working from forwarder to receiver
there is no encryption settings as per the inputs.conf file. i just have the default data in the inputs.conf file when i installed it.
also the forwarders has the data as below
[default]
host = xxxx

[WinEventLog://Application]
disabled = 0
index = xxxx
sourcetype = security

receiver splunkd log is not being updated with senders information

could you please help me in fixing this.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...