F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

ohbuckeyeio — Fri, 19 Feb 2021 21:59:36 GMT

This is a pretty specific use case but was difficult to work through. Documenting for future generations.

Re: F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

ohbuckeyeio — Fri, 19 Feb 2021 22:06:14 GMT

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/"> ProxyPass https://localhost:8000/ ProxyPassReverse https://localhost:8000/ RequestHeader set Remote-User "%{REMOTE_USER}s" SSLRequireSSL AuthType Kerberos AuthName "Kerberos Login at MYCO.COM" KrbAuthRealms MYCO.COM KrbMethodK5Passwd Off Krb5Keytab "/etc/krb5.keytab" Require valid-user KrbMethodNegotiate On KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER #Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs Deny from all Allow from 111.222.333.444 Satisfy any </Location>

topic F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication in Deployment Architecture

F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

Re: F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication