All Apps and Add-ons

Why am I getting "Connection closed by server" errors with a single Splunk instance behind AWS ELB?

New Member

We have single instance, which is on the private subnet and it works really well for all the instances that use its private ip. Then we have created a public ELB, but it is giving the connection refused error:

02-18-2016 18:16:15.628 +0000 INFO  TcpOutputProc - Connection to x.164.x.254.1:9997 closed. Connection closed by server. 02-18-2016 18:16:15.628 +0000 WARN  TcpOutputProc
- Applying quarantine to ip=x.164.x.254.1 port=9997
_numberOfFailures=5 02-18-2016 18:16:15.631 +0000 INFO  TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.859 +0000 INFO  TcpOutputProc - Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.859 +0000 INFO  TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.860 +0000 INFO  TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:16:15.860 +0000 WARN  TcpOutputProc
- Applying quarantine to ip=x.23.x.228 port=9997 _numberOfFailures=5 02-18-2016 18:16:15.860 +0000 INFO  TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:16:45.626 +0000 INFO  TcpOutputProc
- Removing quarantine from idx=x.164.x.254.1:9997 02-18-2016 18:16:45.629 +0000 INFO  TcpOutputProc
- Connected to idx=x.164.x.254.1:9997 02-18-2016 18:16:45.630 +0000 INFO  TcpOutputProc - Connection to x.164.x.254.1:9997 closed. Connection closed by server. 02-18-2016 18:16:45.630 +0000 WARN  TcpOutputProc
- Applying quarantine to ip=x.164.x.254.1 port=9997
_numberOfFailures=6 02-18-2016 18:16:45.632 +0000 INFO  TcpOutputProc
- Connected to idx=x.164.x.254.1:9997 02-18-2016 18:17:15.627 +0000 INFO  TcpOutputProc - Removing quarantine from idx=x.23.x.228:9997 02-18-2016 18:17:15.627 +0000 INFO  TcpOutputProc
- Removing quarantine from idx=x.164.x.254.1:9997 02-18-2016 18:17:15.630 +0000 INFO  TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:17:15.631 +0000 INFO  TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:17:15.631 +0000 WARN  TcpOutputProc
- Applying quarantine to ip=x.23.x.228 port=9997 _numberOfFailures=7 02-18-2016 18:17:15.633 +0000 INFO  TcpOutputProc - Connected to idx=x.23.x.228:9997 02-18-2016 18:17:15.634 +0000 INFO  TcpOutputProc
- Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:17:15.637 +0000 INFO  TcpOutputProc - Connected to idx=x.164.x.254.1:9997

Steps taken so far:
- check the firewall ---------> OK
- telnet to the receiver(indexer) on port 9997 ----------> Successfully connected

my outputs.conf for reference:

[tcpout]
defaultGroup = my_splunk

[tcpout:my_splunk]
autoLB = true
server = mysplunk.test.com:9997

Thanks,

0 Karma

New Member

HI,

I had the same issue.

The splunk server log showed some SSL3 issues.
"routines:SSL3_GET_RECORD:wrong version number" as this was a brand new installation of the forwarder, I had to search Mr.Google for quite a while to find the right answer.

The only kind of workaround I could find was to enable the insecure SSL-Version.

https://answers.splunk.com/answers/552516/ssl-error-after-upgrading-from-661-to-662.html

0 Karma

Builder

Is encryption taking place anywhere? If so, take a look at these and see if they help:
https://answers.splunk.com/answers/28040/forwarder-tcpoutputproc-connection-closed-by-server.html
https://answers.splunk.com/answers/112014/ssl-connection-between-indexer-and-forwarder-validation.ht...
Also, if you're using SSL/TLS, some firewalls are smart enough to require the protocol be set in the rules.

###

If this reply helps you, an upvote would be appreciated.
0 Karma

Motivator

What does your ELB configuration look like? What is your keepalive set to? Keep in mind you need to set the protocol to TCP, this is not an HTTP protocol you are load balancing.

Generally speaking, Splunk advises AGAINST putting a load balancer in front of your indexers.

http://docs.splunk.com/Documentation/Splunk/6.1/Forwarding/Setuploadbalancingd

Note: When implementing load balancing between forwarders and receivers, you must use the forwarder's inherent capability. Do not use an external load balancer. The use of external load balancers between forwarders and receivers will not work properly.

0 Karma