This is the first time I am setting up an AWS Load Balancer for my search heads. We have 4 search heads in our Search Head Cluster. I have configured the ELB, but when I access the DNS, it throws 503 error. It says the connection is refused. Now this search heads are created on AWS also. I would like to know what are changes that we have to make on the Splunk side to access splunkweb on ELB DNS?
Please reply ASAP.
This occurs because Splunk 'thinks' its running on http not https, so its rewriting the URLs to use what it perceives to be the correct scheme - however you can work around this by Enabling TLS on the backend ELB connection too - although this means your doing two lots of encryption/decryption.
There is an existing feature request for this SPL-79993
The ELB setup with a Splunk Search Head Cluster doesn't require any special changes on the Splunk side for access. You will want to verify the following setup steps on the ELB and Security Groups are correct:
One possible issue can be that the ELB health check is failing and taking your Splunk Search Heads out of service. If you are using an HTTP health check with a ping target of HTTP:8000/, this check on the Splunk Search Head will fail since the ELB Health Check is expecting a 200 response code and the Splunk Search Head will actually return a 303 redirect response instead. The Splunk Search Head URL is typically http://hostname:8000/en-US/account/login?return_to=%2Fen-US%2F so the health check ping target would need to be HTTP:8000/en-US/account/login?return_to=%2Fen-US%2F for it to work properly.
One setting for the ELB and all other load balancers with regards to Search Head Clustering is that you need to enable "sticky" or "persistent" connections so that a user remains on a single search head during their session. Here are more details regarding this setting: http://docs.splunk.com/Documentation/Splunk/6.3.2/DistSearch/UseSHCwithloadbalancers
i have to say this is not working with Splunk 7.0.0 enterprise search head cluster.
we have set up ELB in front of the search head nodes. ELB is listening on port 443 and forward to port 8000 on backend search head.
I did some test and found that if you try to access HTTPS, the backend will do a 303 redirect to HTTP. For example, if you access https://splunk.example.com, backend server will 303 redirect it to http://splunk.example.com. So if the ELB has no port 80 listener, it will failed with timed out. If the ELB has port 80 listener, eventually you will be redirected to HTTP url. Then, there's nothing happening with HTTPS, it was just skipped and ignored.
I really don't understand why backend search head server do a 303 redirect on HTTPS request and there were a lot of discussion but none of them giving a solution, all ended up with nothing.
Please, someone who had same issue here, post your answer here. Splunk has a really bad community ecosystem compared to AWS. Hope some expert can help here.
Hi, Can you please share the specific settings that we need to do on Amazon ELB to enable the "sticky" / "persistent" settings?
We tried the following but it didn't work -