Splunk behind a reverse proxy refuses to stay on https

[bobbycrispbox]

This is a problem I've been battling with for ages, and I feel like I've looked at every related splunk>answers post, I've Googled extensively for Apache, Splunk and CherryPy answers, and I just can't get it sorted.

The set-up is this: My splunk (free) instance sits on a server which also has an apache2 instance. The Splunk server listens only on the loopback address, at http://127.0.0.1:8000/. My apache server is taking the requests from wherever they come at https://splunk.mydomain.com/ and ProxyPasses them on to the local Splunk address.

Whenever I connect to my apache url, on https, the redirects (from / to /en-GB/, from /en-GB/ to /en-GB/app/launcher/ etc) every redirect comes back as http:// instead of https://.

I've tried so many different Apache directives, including:

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-Forwarded-Ssl "on"
RequestHeader set X-Url-Scheme "https"
SetEnvIf X-Url-Scheme https HTTPS=1

and none seem to resolve the issue.
I've tried all sorts of variations in web.conf

enableSplunkWebSSL = true | false
tools.proxy.on = true | false

but none have led me to a solution.
In the end I've implemented a new VirtualHost listening on port 80 which forwards requests from http to https, but it's a horrible hack and a waste of system resources.

Can anyone tell me their recipes for putting an Apache reverse proxy with SSL in front of a Splunk instance running on the same server?

Current Apache configuration:

<VirtualHost _default_:443>
        ServerName splunk.mydomain.com
        ServerAlias splunk.alias.mydomain.com
        CustomLog ${APACHE_LOG_DIR}/access-splunk.mydomain.com.log combined
        RequestHeader set X-Forwarded-Proto "https"
        RequestHeader set X-Forwarded-Port "443"
        RequestHeader set X-Forwarded-Ssl "on"
        RequestHeader set X-Url-Scheme "https"
        <IfModule mod_proxy.c>
                ProxyPass / http://127.0.0.1:8000/
                ProxyPassReverse / http://127.0.0.1:8000/
                SetEnvIf X-Url-Scheme https HTTPS=1
        </IfModule>
</VirtualHost>

Current web.conf

[settings]
enableSplunkWebSSL = false

[bobbycrispbox]

No, the workaround remains in place

[jtacy · Accepted solution]

It's a good question, but without documented support for SSL offload I've always found it safest to run a bridged config where both your web server and Splunk will talk HTTPS.

Apache config

<VirtualHost _default_:443>
    ServerName splunk.mydomain.com
    ServerAlias splunk.alias.mydomain.com
    CustomLog ${APACHE_LOG_DIR}/access-splunk.mydomain.com.log combined

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/xyzzy.crt
    SSLCertificateKeyFile /etc/pki/tls/private/xyzzy.key
    SSLCertificateChainFile /etc/pki/tls/certs/interm.crt

    SSLProxyEngine on
    <IfModule mod_proxy.c>
        ProxyPass / https://127.0.0.1:8443/ retry=0
        ProxyPassReverse / https://127.0.0.1:8443/
    </IfModule>
</VirtualHost>

web.conf

[settings]
httpport = 8443
enableSplunkWebSSL = 1

That's essentially working config with Splunk Enterprise but I haven't tested with Splunk Free. It would be nice if someone from Splunk commented on whether and how SSL offload is supported with the product. Details like secure cookies should be accounted for in a correct solution. I imagine it would be hard to justify SSL offload from a performance standpoint but it can make troubleshooting easier and simplify traffic inspection requirements.

[bobbycrispbox]

Thanks to jtacy for your assistance with this. I have not resolved my specfic issue but your answers have given me confidence that I'm doing the right things in my configuration. The problem probably lies elsewhere in the apache config. Until I work out what it is, I'll continue to use my additional virtualhost redirect from port 80 to port 443.

[burwell]

Hi. Did you ever solve this problem?

[bobbycrispbox]

Thanks jtacy for your response.

I've switched over to HTTPS on Splunk web but the problem persists.
Apache config:

<VirtualHost _default_:443>
    ServerName splunk.mydomain.com
    ServerAlias splunk.alias.mydomain.com
    CustomLog ${APACHE_LOG_DIR}/access-splunk.mydomain.com.log combined
    SSLProxyEngine on
    <IfModule mod_proxy.c>
        ProxyPass / https://127.0.0.1:8443/ retry=0
        ProxyPassReverse / https://127.0.0.1:8443/
    </IfModule>
</VirtualHost>

(note that the certificate settings are there, just not in this part of the apache config)
web.conf:

[settings]
enableSplunkWebSSL = true
httpport = 8443

Are there any other places I should look for misconfigured settings?

[jtacy]

Hmm...I changed to a Splunk Free license and it still worked fine so it doesn't look like there's anything unusual/broken about HTTPS in Free.

1. Does Splunk definitely work properly if you hit it directly with a browser on port 8443?
2. If you run curl -v -k -o /dev/null https://localhost or another HTTP get tool on the server that hits Apache what do you get? There should be a line that says something like Location: https://localhost/en-US/.
3. You mentioned that you have some other Apache config out there. Any mod_rewrite or Redirect entries in there? This config tests fine when added to the base config that ships with RHEL6 and Splunk 6.3.3 so I'm puzzled.

[bobbycrispbox]

Thanks again jtacy

1. Yes, works perfectly when I hit https://172.23.9.2:8443/

2. I had to modify the curl command a bit to get what you wanted (had to pass a host header as it's a name-based virtual host), here's the result

   root@debian:~# curl -v -k -o /dev/null -H 'Host: splunk.mydomain.com' https://172.23.9.2
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Error reading ca cert path /etc/ssl/certs - PolarSSL: (-0x2900) X509 - Read/write of file failed

   GET / HTTP/1.1
   User-Agent: curl/7.40.0
   Accept: /
   Host: splunk.mydomain.com

   < HTTP/1.1 303 See Other
   < Date: Fri, 10 Feb 2017 01:07:00 GMT
   < Server: Splunkd
   < Content-Type: text/html; charset=UTF-8
   < X-Content-Type-Options: nosniff
   < Content-Length: 575
   < Location: http://splunk.mydomain.com/en-US/
   < Vary: Accept-Language,Accept-Encoding
   < X-Frame-Options: SAMEORIGIN
   <
   { [575 bytes data]
   100 575 100 575 0 0 1704 0 --:--:-- --:--:-- --:--:-- 1706
   root@debian:~#

3. It's a multi-purpose server (a bit of a sandbox) and I'm coming to the conclusion that there's something about its setup that is causing these problems. I can say that the VirtualHost is clean and has no specific rewrites or redirects.