Deployment Architecture

What are the configuration changes we have to make in order to load balance clustered Search Heads using AWS Elastic Load Balancer?

Engager

Hi,

This is the first time I am setting up an AWS Load Balancer for my search heads. We have 4 search heads in our Search Head Cluster. I have configured the ELB, but when I access the DNS, it throws 503 error. It says the connection is refused. Now this search heads are created on AWS also. I would like to know what are changes that we have to make on the Splunk side to access splunkweb on ELB DNS?

Please reply ASAP.

Thanks!

Ultra Champion

This occurs because Splunk 'thinks' its running on http not https, so its rewriting the URLs to use what it perceives to be the correct scheme - however you can work around this by Enabling TLS on the backend ELB connection too - although this means your doing two lots of encryption/decryption.

There is an existing feature request for this SPL-79993
See:
https://answers.splunk.com/answers/103674/using-aws-https-elb-with-ec2-splunk-web-on-http-port-8000....
https://answers.splunk.com/answers/327618/how-to-configure-a-splunk-623-search-head-cluster.html

0 Karma

Splunk Employee
Splunk Employee

The ELB setup with a Splunk Search Head Cluster doesn't require any special changes on the Splunk side for access. You will want to verify the following setup steps on the ELB and Security Groups are correct:

  1. Configure all the proper Security Groups to allow the ELB to communicate with the Search Heads within your VPC. Here are the recommended Security Group rules for the ELB with back-end instances such as Splunk Search Heads (http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-security-groups.html#recom...)
  2. Configure the ELB listener to the proper web port for your back-end instance (aka Splunk Search Head). The default web port for a Splunk Search Head is 8000, but this port can be changed on Splunk if needed.
  3. Verify that the health check between the ELB and the Splunk Search Heads is working and that the Splunk Search Heads are 'InService'.

One possible issue can be that the ELB health check is failing and taking your Splunk Search Heads out of service. If you are using an HTTP health check with a ping target of HTTP:8000/, this check on the Splunk Search Head will fail since the ELB Health Check is expecting a 200 response code and the Splunk Search Head will actually return a 303 redirect response instead. The Splunk Search Head URL is typically http://hostname:8000/en-US/account/login?return_to=%2Fen-US%2F so the health check ping target would need to be HTTP:8000/en-US/account/login?return_to=%2Fen-US%2F for it to work properly.

One setting for the ELB and all other load balancers with regards to Search Head Clustering is that you need to enable "sticky" or "persistent" connections so that a user remains on a single search head during their session. Here are more details regarding this setting: http://docs.splunk.com/Documentation/Splunk/6.3.2/DistSearch/UseSHCwithloadbalancers

New Member

Hi,

i have to say this is not working with Splunk 7.0.0 enterprise search head cluster.
we have set up ELB in front of the search head nodes. ELB is listening on port 443 and forward to port 8000 on backend search head.
I did some test and found that if you try to access HTTPS, the backend will do a 303 redirect to HTTP. For example, if you access https://splunk.example.com, backend server will 303 redirect it to http://splunk.example.com. So if the ELB has no port 80 listener, it will failed with timed out. If the ELB has port 80 listener, eventually you will be redirected to HTTP url. Then, there's nothing happening with HTTPS, it was just skipped and ignored.
I really don't understand why backend search head server do a 303 redirect on HTTPS request and there were a lot of discussion but none of them giving a solution, all ended up with nothing.

Please, someone who had same issue here, post your answer here. Splunk has a really bad community ecosystem compared to AWS. Hope some expert can help here.

Thanks

0 Karma

Path Finder

Hi, Can you please share the specific settings that we need to do on Amazon ELB to enable the "sticky" / "persistent" settings?

We tried the following but it didn't work -

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. On the navigation pane, under LOAD BALANCING, choose Load Balancers.
  3. Select your load balancer.
  4. On the Description tab, choose Edit stickiness.
  5. On the Edit stickiness page, select Enable load balancer generated cookie stickiness.
  6. Leave the Expiration Period blank, so that by default the sticky session lasts for the duration of the browser session.
  7. Choose Save.
0 Karma