How to configure a Splunk 6.2.3 search head cluster behind an AWS Elastic Load Balancer (ELB) to terminate SSL?

[jnicholsenernoc]

We are running 6.2.3 and are using search head clustering.

We would like to use an AWS ELB to terminate SSL, and then send the data to port 8000 on the search head nodes. The problem is that Splunk Web will redirect to HTTP and URLs in the UI will be plain HTTP and not HTTPs.

Is it possible to configure Splunk so that it is aware it is behind an SSL proxy?

[awurster]

@jnicholsenernoc and all - here is my working solution. we use CloudFormation and ansible to deploy everything pretty nicely in about 5-10 minutes. i've had mixed results trying to use Splunk's CFN and ansible tasks for several reasons - so i've rolled my own.

i'm a hands on guy - so here's a code snippet for your reference:
https://bitbucket.org/snippets/asecurityteam/58G8X

there are a few caveats worth explaining though about our setup:

1. we build our search heads and ELBs using CFN - meaning we don't point and click anything in the AWS Console
2. the certificate itself is installed already, so in the CFN snippet { "Ref" : "SplunkSearchLoadBalancerSSLCert" } refers to the cert's ARN
3. the search head instances themselves run SSL on port 8443 so that we don't need to deploy or maintain certs, and also we don't need to run splunk as a privileged user (to use ports < 1024 in linux; you could use iptables or apache or something to redirect on 443 but meh?)
4. nodes themselves use self-signed certs because it's easier to manage / irrelevant for the most part, and besides the trusted cert sits on the ELB
5. nodes themselves are in an Auto Scale Group - cause it's easier to build new ones and tie in easily to the ELB all at once... but that shouldn't be a requirement either way
6. the reason why we're not overly concerned about what port to run or what cert to use on the individual nodes is that we use security groups to control access to the nodes versus the ELBs. you literally can't log in to the search heads individually - only access via the ELB. this line controls the ELB access, and another one elsewhere in our CFN stack does a similar thing for the Search Head cluster members "SecurityGroups" : [ { "Ref" : "SplunkSearchLoadBalancerSG" } ]
7. i'm not 100% sure this is the best / right LB policy (ie caching, timeouts, retries, etc), so there could be better ways

[Lowell · Accepted solution]

Not supported. You can do HTTPS -> HTTPS with the ELB, as far as I know that's still the best option. Still waiting on SPL-79993.

https://answers.splunk.com/answers/103674/using-aws-https-elb-with-ec2-splunk-web-on-http-port-8000....

[lycollicott]

I am no load balancer expert, but we had exactly the same behaviour with an F5 and our LB engineer enabled a setting to "rewrite_all" of the redirects. It sounds like you could do the same on your LB.

[awurster]

I originally downvoted it to try and simply even the score back to 0, but it went to -1...

but more importantly because i felt the this post didn't directly address the AWS focus of the question. and as @jnicholsenernoc mentioned there is no option for rewriting. even worse - splunk doesn't give a nice "how to" guide for AWS so it can be quite daunting to set it up if you're an AWS newbie. like F5's - AWS has a number of options you have to tune the LB group which can be rather confusing.

so the conjecture in this case while well-intentioned, was ultimately not useful.

[lycollicott]

Gee, load balancer question. Load balancer answer. Actually, a correct answer for many LB technologies, but whatever.

[jnicholsenernoc]

Unfortunately, there isn't a "rewrite_all" option for an Amazon ELB like there is for an F5 LB.