Deployment Architecture

F5 Health Monitor of Splunk when Apache sits in front of Splunk for Kerberos Authentication

ohbuckeyeio
Path Finder

This is a pretty specific use case but was difficult to work through.  Documenting for future generations.

Labels (1)
0 Karma
1 Solution

ohbuckeyeio
Path Finder

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/">
ProxyPass https://localhost:8000/
ProxyPassReverse https://localhost:8000/
RequestHeader set Remote-User "%{REMOTE_USER}s"

SSLRequireSSL
AuthType Kerberos
AuthName "Kerberos Login at MYCO.COM"
KrbAuthRealms MYCO.COM
KrbMethodK5Passwd Off
Krb5Keytab "/etc/krb5.keytab"
Require valid-user
KrbMethodNegotiate On
KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER

#Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs
Deny from all
Allow from 111.222.333.444
Satisfy any

</Location>

View solution in original post

0 Karma

ohbuckeyeio
Path Finder

I have an F5 Health Monitor in place to determine when the Splunk service backed by Apache/Kerberos on a search head has dropped. This will pull those members out of the pool and prevent connections from heading that way.

The changes:

In F5, I created a new health monitor called Splunk_Apache_https_monitor. This monitor sends a HEAD request to the pool members to test the connection. This is the header I built for that HEAD request:

HEAD / HTTP/1.1\r\nHost:myco.com\r\nUser-agent: MYCO_F5_User_Agent

The HTTP 1.1 standard calls for a header that has a host and user-agent directive, but they don’t really have to mean anything in our configuration to pass the check. I made them agnostic with a host of “myco.com” and User-agent of “MYCO_F5_User_Agent” so we know where it’s coming from and can apply it to all the things.

The monitor checks the response from the header request and this regex parses for any service interrupting http status codes that might arise.
HTTP/1\.[01] [2-4]0[0-6]

You can see these F5 requests in the /etc/httpd/logs/splunkweb-access_log.

This is a health check that sees the backend of Splunk is active:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 303 - "-" "MYCO_F5_User_Agent"

This is a health check that reports Splunk is down, but Apache is running:
111.222.333.444 - - [18/Feb/2021:13:47:41 -0800] "HEAD / HTTP/1.1" 503 - "-" "MYCO_F5_User_Agent"

On the Splunk side, I modified the /etc/httpd/conf.d/splunkweb.conf file’s <Location “/” > directive to bypass the kerberos request for the F5 IP addresses. See below.

<Location "/">
ProxyPass https://localhost:8000/
ProxyPassReverse https://localhost:8000/
RequestHeader set Remote-User "%{REMOTE_USER}s"

SSLRequireSSL
AuthType Kerberos
AuthName "Kerberos Login at MYCO.COM"
KrbAuthRealms MYCO.COM
KrbMethodK5Passwd Off
Krb5Keytab "/etc/krb5.keytab"
Require valid-user
KrbMethodNegotiate On
KrbLocalUserMapping On #removes @MYCO.COM from REMOTE_USER

#Only allow users through who provide a kerberos ticket, but igore this rule for F5 IPs
Deny from all
Allow from 111.222.333.444
Satisfy any

</Location>
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...