Installation
Highlighted

Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Path Finder

I am having some issues specifically with the Splunk 6.6.0 version and my AWS ELB health checks not going healthy. I wanted to see if it is a one-off issue or others were having the same problems before I open up a Splunk Enterprise Support Case.

The problem, I have a proof of concept environment setup within one of our AWS accounts and recently upgraded it from v6.5 to v6.6.0 to test it out before deploying it. Post upgrade, the following health check, which were were working fine prior to the upgrade, is no longer working.

alt text

I have attempted to remove the nodes from the original ELB and add them back into it without any luck. I have also deleted the original ELB and re-created it with the same settings as before the upgrade without any luck.

There are only two ways I can get the health check to work properly. The first one is when I change the health check over to TCP:443 instead of HTTPs:443 and the nodes flip over to inservice. That is not an option I want to use as it only watches for a listening port and not that Splunk is running. The second one is if I put Splunk v6.3 or v6.5 instances into the same ELB and those nodes will flip over to inservice.

As a side note, the exact same health checks works fine in a Application ELB but not with the Classic ELB. The problem with that option is we cannot get it working for the Splunk API, another project for later.

Any thoughts? Thanks in advanced for the help!

Labels (1)
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Communicator

Splunk removed the TLS1.2 cypher from web.conf, which breaks the ELB health check and SSL termination. Not sure if it's something that AWS needs to fix as well (as in support the stronger cyphers on the backend SSL connections), but in the meantime, add the following to your local web.conf in the location of your choice

local/web.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:TLSv1.2+HIGH:@STRENGTH

6.5.x setting:

/opt/splunk/etc/system/default/web.conf cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

6.6.x setting:
/opt/splunk/etc/system/default/web.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

View solution in original post

Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Contributor

Hi vliggio, thanks it solved our problems here

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Path Finder

Thanks dude, I spent a whole day until a friend send me this link. works fine on splunkweb 6.6.2.

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Explorer

Thank you! I've stuck on this problem for a while until I saw this answer. my splunk is upgrade from 6.5.2 to 6.6.2, now it works like a charm 🙂

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Motivator

Thank you for this!

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

New Member

Thank You

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Path Finder

Hello,

We are connecting to port 8000.

Listener is configured with following:
LB:HTTPS LB port:443 inst protocol:HTTPS Inst port:8000
Cipher is set to "predefined security policy"

Could Please let me know what is wrong with the configuration.
web.conf:
[settings]
httpport = 8000
enableSplunkWebSSL = true
splunkdConnectionTimeout = 60

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

regards,
Bhasker.K

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Motivator

Your ELB is listening on 443.

You are hitting your ELB on 8000.

Consider that carefully. 🙂

0 Karma
Highlighted

Re: Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

Path Finder

Hello,

we ran into similar problem, we are using classic ELB and splunk 7.x.
We are able to connect to 8000 if https://privateIp:8000//en-US/account/login?return_to=%2Fen-US%2F, this works. But if I use the https://ELB:8000//en-US/account/login?return_to=%2Fen-US%2F it doesn't work.
Can any one please help us what exactly is the problem. Appreciate your help.

when I do a curl on ELB:
HTTP/1.1 503 Service Unavailable: Back-end server is at capacity
Connection: keep-alive

Below is the web.conf:

httpport = 8000
enableSplunkWebSSL = true
splunkdConnectionTimeout = 60

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

0 Karma