Deployment Architecture

F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

ohbuckeyeio
Path Finder

This is a pretty specific use case but was difficult to work through.  Documenting for future generations.

Labels (1)
0 Karma
1 Solution

ohbuckeyeio
Path Finder

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/">
ProxyPass https://localhost:8000/
ProxyPassReverse https://localhost:8000/
RequestHeader set Remote-User "%{REMOTE_USER}s"

SSLRequireSSL
AuthType Kerberos
AuthName "Kerberos Login at MYCO.COM"
KrbAuthRealms MYCO.COM
KrbMethodK5Passwd Off
Krb5Keytab "/etc/krb5.keytab"
Require valid-user
KrbMethodNegotiate On
KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER

#Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs
Deny from all
Allow from 111.222.333.444
Satisfy any

</Location>

View solution in original post

0 Karma

ohbuckeyeio
Path Finder

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/">
ProxyPass https://localhost:8000/
ProxyPassReverse https://localhost:8000/
RequestHeader set Remote-User "%{REMOTE_USER}s"

SSLRequireSSL
AuthType Kerberos
AuthName "Kerberos Login at MYCO.COM"
KrbAuthRealms MYCO.COM
KrbMethodK5Passwd Off
Krb5Keytab "/etc/krb5.keytab"
Require valid-user
KrbMethodNegotiate On
KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER

#Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs
Deny from all
Allow from 111.222.333.444
Satisfy any

</Location>
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...