Why are AWS ELB Health Checks not working properly after upgrading to Splunk 6.6.0?

[eandresen]

I am having some issues specifically with the Splunk 6.6.0 version and my AWS ELB health checks not going healthy. I wanted to see if it is a one-off issue or others were having the same problems before I open up a Splunk Enterprise Support Case.

The problem, I have a proof of concept environment setup within one of our AWS accounts and recently upgraded it from v6.5 to v6.6.0 to test it out before deploying it. Post upgrade, the following health check, which were were working fine prior to the upgrade, is no longer working.

I have attempted to remove the nodes from the original ELB and add them back into it without any luck. I have also deleted the original ELB and re-created it with the same settings as before the upgrade without any luck.

There are only two ways I can get the health check to work properly. The first one is when I change the health check over to TCP:443 instead of HTTPs:443 and the nodes flip over to inservice. That is not an option I want to use as it only watches for a listening port and not that Splunk is running. The second one is if I put Splunk v6.3 or v6.5 instances into the same ELB and those nodes will flip over to inservice.

As a side note, the exact same health checks works fine in a Application ELB but not with the Classic ELB. The problem with that option is we cannot get it working for the Splunk API, another project for later.

Any thoughts? Thanks in advanced for the help!

[kevinduterne]

ECDHE are not supported for AWS ELB classic loadbalancers healthchecks

[basu42002]

When you upgrade to 7.x, we need to update the configuration with new cipher suites. I was able to finally get the health checks working.
change the cipher suite accepted by your server to the ones which are offered by the ELB in the ClientHello. To do this you can update your configuration file to contain this:
cipherSuite = AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-SHA:RC4-SHA:DES-CBC3-SHA:DES-CBC-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA

This solution gives broader spectrum of ciphers for the server to pick from.

[vliggio · Accepted solution]

Splunk removed the TLS1.2 cypher from web.conf, which breaks the ELB health check and SSL termination. Not sure if it's something that AWS needs to fix as well (as in support the stronger cyphers on the backend SSL connections), but in the meantime, add the following to your local web.conf in the location of your choice

local/web.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:TLSv1.2+HIGH:@STRENGTH

6.5.x setting:

/opt/splunk/etc/system/default/web.conf cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

6.6.x setting:
/opt/splunk/etc/system/default/web.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

[Kzark]

This is absolutely the solution. Spent hours looking for the answer to why SSL through the AWS ELBs would not work and this resolves it. Should be included in documentation somewhere.

[basu42002]

Hello,

we ran into similar problem, we are using classic ELB and splunk 7.x.
We are able to connect to 8000 if https://privateIp:8000//en-US/account/login?return_to=%2Fen-US%2F, this works. But if I use the https://ELB:8000//en-US/account/login?return_to=%2Fen-US%2F it doesn't work.
Can any one please help us what exactly is the problem. Appreciate your help.

when I do a curl on ELB:
HTTP/1.1 503 Service Unavailable: Back-end server is at capacity
Connection: keep-alive

Below is the web.conf:

httpport = 8000
enableSplunkWebSSL = true
splunkdConnectionTimeout = 60

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

[emiller42]

Since the previous was edited:

What is the listener configuration on the ELB? a 503 implies that there is either no listener on 8000, or the listener has no accessible backend. Can you verify the listener is set up as HTTPS 8000 -> HTTPS 8000?

[basu42002]

ELB listener:
Load Balancer Protocol
Load Balancer Port
Instance Protocol
Instance Port
Cipher
SSL Certificate
HTTPS 443 HTTPS 8000 Change

(IAM) Change

Do we need to make changes to ELB port, currently its 443 and instance port is 8000.

[emiller42]

You could change the ELB listener port.

Or you could just use https://ELB/en-US/account/login

If your ELB is listening on 443, send your requests to 443. 🙂

[basu42002]

I did try with https://myelb.com (there is no page error, its just blank) and also with /en-US/account/login, no luck.
Also I have taken tcpdump and verified if ELB sent "Client Hello" packets, no such packets found in the dump.

Below is the health check configuration:
Ping Target
HTTPS:8000/en-US/account/login?return_to=%2Fen-US%2F
Timeout 10 seconds
Interval 30 seconds
Unhealthy threshold 2
Healthy threshold 3

[emiller42]

When you look at the instances tab of your load balancer, is the instance listed? What is it's Status?

[basu42002]

on ELB->instances->Status is "out of service", instance is listed and we have splunk search node running,
and I am able to access it using https://privateIP:8000, but when I use https://myELB.com it doesn't work, also the status is out of service as mentioned earlier.

I have also reached out to AWS support but yet to get response from them.

[emiller42]

I assume you're in a VPC. Do you have security groups set up to allow traffic from the ELB to your instance on port 8000?

[basu42002]

Yes we are in a VPC, security groups are setup to allow traffic from ELB.
"All traffic" is allowed internally and it worked when the listener was configured as http with ELB->port 80->http splunk->8000

Then I have added the following in web.conf:
enableSplunkWebSSL = true

and changed the listener to to ELB https->port 443->splunk https->port 8000

Any other suggestions please, I am stuck as the instance status is out of service and not sure why ELB is not able to ping the backend instance.

[emiller42]

Thank you for this!

[basu42002]

Hello,

We are connecting to port 8000.

Listener is configured with following:
LB:HTTPS LB port:443 inst protocol:HTTPS Inst port:8000
Cipher is set to "predefined security policy"

Could Please let me know what is wrong with the configuration.
web.conf:
[settings]
httpport = 8000
enableSplunkWebSSL = true
splunkdConnectionTimeout = 60

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

regards,
Bhasker.K

[emiller42]

Your ELB is listening on 443.

You are hitting your ELB on 8000.

Consider that carefully. 🙂

[syadavsplunk]

Thank You

[walker_liu]

Thank you! I've stuck on this problem for a while until I saw this answer. my splunk is upgrade from 6.5.2 to 6.6.2, now it works like a charm 🙂

[freaklin]

Thanks dude, I spent a whole day until a friend send me this link. works fine on splunkweb 6.6.2.

[fabiocaldas]

Hi vliggio, thanks it solved our problems here