F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

[ohbuckeyeio]

This is a pretty specific use case but was difficult to work through.  Documenting for future generations.

[ohbuckeyeio · Accepted solution]

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/">
ProxyPass https://localhost:8000/
ProxyPassReverse https://localhost:8000/
RequestHeader set Remote-User "%{REMOTE_USER}s"

SSLRequireSSL
AuthType Kerberos
AuthName "Kerberos Login at MYCO.COM"
KrbAuthRealms MYCO.COM
KrbMethodK5Passwd Off
Krb5Keytab "/etc/krb5.keytab"
Require valid-user
KrbMethodNegotiate On
KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER

#Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs
Deny from all
Allow from 111.222.333.444
Satisfy any

</Location>