Deployment Architecture

Fixing 502 errors when front-ending Search Heads with an AWS application load balancer?

Splunk Employee
Splunk Employee

We have a Splunk deployment in AWS and have our Search Head Cluster front-ended with an ALB (not ELB). Users frequently have the screen say "502 bad gateway", which usually goes away after a refresh or two. Has anyone else seen this, and figured out how to fix it?

1 Solution

Explorer

I've been able to resolve this issue by disabling HTTP/2 on our ALB. We're running Splunk Enterprise 7.1.0 and were seeing 73286 requests per hour of which 1205 were ELB 502 errors (0,16%). Directly after disabling HTTP/2 on the ALB, we are seeing 0 ELB 502 errors.

Next question is: Why does it break?

View solution in original post

Explorer

I've been able to resolve this issue by disabling HTTP/2 on our ALB. We're running Splunk Enterprise 7.1.0 and were seeing 73286 requests per hour of which 1205 were ELB 502 errors (0,16%). Directly after disabling HTTP/2 on the ALB, we are seeing 0 ELB 502 errors.

Next question is: Why does it break?

View solution in original post

Path Finder

This worked for me.

0 Karma

Splunk Employee
Splunk Employee

My customer can validate this has also worked for them. Great find @joeydenbroeder

0 Karma

Explorer

how to check this one?

0 Karma

Path Finder

@akira were you able to fix this? I'm having the same issue.

0 Karma

Explorer

I'm seeing the same situation after migrating to AWS.

We're running Splunk Enterprise 7.0.2.

3 search heads, clustered behind an ALB. We see about 31468 requests per hour. 217 of those are 5XX errors on the ELB. (0,07%)

We are also seeing that after we run a ( successful ) search, when it's done and settled a "server error" message appears below the query bar. All results are there, and the page works fine but it is odd.

0 Karma

New Member

I am having the same issue , the only thing is I am not even able to see a successful splunkweb page at all.
I have configured my environment using the ALB and 3 search heads behind it.

so User browser(https) ---> ALB listens on 443 ---> Forward to Target Group which has protocol for HTTPS and Port 8000 for backend servers.

all search heads are configured to use https but it just gives me 502 Bad Gateway all the time. I enabled the access logs to ALB and here's what I get.

h2 2018-03-19T17:34:59.318960Z app/Splunk-SearchHead-ELB/d24e3730216c0f34 37.228.224.60:34058 10.11.2.83:8000 -1 -1 -1 502 - 95 208 "GET https://splunk-searchhead-elb-985980458.us-east-1.elb.amazonaws.com:443/en-US/static/@01A10D5DE1BF7B... HTTP/2.0" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:542993520366:targetgroup/SHTargetGroup/4afe5809d39a7bac "Root=1-5aaff4c3-87775f57ad096cf7cad703d8" "splunk-searchhead-elb-985980458.us-east-1.elb.amazonaws.com" 
0 Karma

Motivator

Hi @akira,

We have used this in our environment and it works fine. I think the issue is with AWS configuration. You should start with your network configuration on AWS

0 Karma