Splunk CAC Based Authentication

[partom24]

Hello All!

Trying to set up CAC Based Auth for SPLUNK 9.1.1 on Windows Server 2022 for the first time. I have successfully setup LDAP and am able to sign into Splunk using an AD username/password without any issues. When I add in the requiredClientCert, enableCertBasedAuth and certBasedUserAuthMethod stanzas, and attempt to access the Splunk GUI, all users are immediately greeted with an 'Unauthorized' message. I've been fighting this for about a week now, and Splunk support hasn't been able to help me pin this down yet. Any assistance would be greatly appreciated.

I've ensured TLS 1.2 registry keys exist in SCHANNEL to Enable TLS 1.2.

Corresponding logs from splunkd.log for the logon attempt are:

09-29-2023 09:02:43.191 -0400 INFO  AuthenticationProviderLDAP [12404 TcpChannelThread] - Could not find user=" \x84\x07\xd8\xb6\x05" with strategy="123_LDAP"
09-29-2023 09:02:43.192 -0400 ERROR HTTPAuthManager [12404 TcpChannelThread] - SSO failed - User does not exist:  \x84\x07\xd8\xb6\x05
09-29-2023 09:02:43.192 -0400 ERROR UiAuth [12404 TcpChannelThread] - user= \x84\x07\xd8\xb6\x05 action=login status=failure reason=sso-failed useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" clientip=<ip>
09-29-2023 09:03:10.247 -0400 ERROR UiAuth [12404 TcpChannelThread] - SAN OtherName not found for configured OIDs in client certificate
09-29-2023 09:03:10.247 -0400 ERROR UiAuth [12404 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

authentication.conf:

[splunk_auth]
minPasswordLength = 8
minPasswordUppercase = 0
minPasswordLowercase = 0
minPasswordSpecial = 0
minPasswordDigit = 0

[authentication]
authSettings = 123_LDAP
authType = LDAP

[123_LDAP]
SSLEnabled = 1
anonymous_referrals = 0
bindDN = CN=<Account>,OU=Service Accounts,OU=<Command Accounts>,DC=<Command>,DC=NAVY,DC=MIL
bindDNpassword = <removed>
charset = utf8
emailAttribute = mail
enableRangeRetrieval = 0
groupBaseDN = OU=SPLUNK Groups,OU=Groups,DC=<command>,DC=NAVY,DC=MIL
groupMappingAttribute = dn
groupMemberAttribute = member
groupNameAttribute = cn
host = DC.<Command>.NAVY.MIL
nestedGroups = 1
network_timeout = 20
pagelimit = -1
port = 636
realNameAttribute = displayName
sizelimit = 1000
timelimit = 15
userBaseDN = OU=Users,OU=<Command Accounts>,DC=<Command>,DC=NAVY,DC=MIL
userNameAttribute = userprincipalname

[roleMap_LDAP]
admin = SPLUNK AUDITOR
can_delete = SPLUNK AUDITOR
network = SPLUNK NETWORK
user = SPLUNK AUDITOR;SPLUNK USERS

web.conf

[settings]
enableSplunkWebSSL = true
privKeyPath = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_key.pem
serverCert = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_server.pem
sslPassword = <removed>
requireClientCert = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\dodCerts\DoDRootCA3.pem
enableCertBasedUserAuth=true
SSOMode=permissive
trustedIP = 127.0.0.1
certBasedUserAuthMethod=PIV

server.conf

[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\dodCerts\DoDRootCA3.pem
serverCert = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_server.pem
sslPassword = <removed>
cliVerifyServerName = true
sslVersions = tls1.2
sslVerifyServerCert = true

[general]
serverName = SPKVSPLUNK2
pass4SymmKey = <removed>
trustedIP = 127.0.0.1

[SPL_Dummy]

Following for I need to do this soon as well... hope you figure it out so I can 😉

[jnoose]

Anyone, anyone? Bueller?

I feel like I'm so close to making this work as well. SSL/TLS is configured, Splunk Web GUI prompts for PIV token + PIN, but it fails out to some "ERROR: Unauthorized" xml garbage in the browser. 

Tailing splunkd for CertBasedUserAuth reveals: error fetching username from client certificate. 

Relevant Settings: 
certBasedUserAuthMethod = PIV

certBasedUserPivOidList = 1.3.6.1.4.1.311.20.2.3,Microsoft Universal Principal Name

Any ideas? 

[jencot01]

Have you tried replacing PIV with EDIPI?  

certBasedUserAuthMethod = EDIPI

[jnoose]

No dice.

Previous errors with PIV/OID were 

ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate

ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

New error with EDIPI

ERROR UiAuth [2488903 TcpChannelThread] - user=<DoDID#> action=login status=failure reason=sso-failed useragent=<browser stuff>

[jencot01]

I just noticed one of your last posts showing the following errors:

Previous errors with PIV/OID were 

ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate

ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

The PIV pulls from the Subject Alternate Name (SAN) "Other Name."  Validate on your PIV the value of Other Name in Subject Alternate Name.  I'm assuming the value you would like to pull is not found in that location.  You will need to find the OID value for the location on your PIV and change the certBasedUserAuthPivOidList values to match the correct location on your PIV. 

Hope this helps.

[jnoose]

Great catch. I noticed this as well and thought I had a smoking gun in 90Meter Smart Card Manager for my token. I noticed: 

Extension Type: Subject Alternative Name
       Oid: 2.5.29.17
              Other Name: Principal Name=<DoD-ID>.ADMN@smil.mil

I put that Oid into the web.conf, restarted, and got the same UiAuth erros 😞

[jencot01]

If you are having the same issue next week, I will be in an environment that I can help better.  I'm currently going off of memory.  Please let me know if you still need help on Monday and I can help troubleshoot better.  Sorry I can't think of anything else to suggest right now.

[jnoose]

I'm going to try and push through today but I reckon I'll be in the same place Monday. I really appreciate your help so far and would be so grateful if dig in deeper next week. Thanks again and enjoy your weekend!

[jnoose]

I'll give it go. PIV just seems like the way to go because my UPN is <myDoDID#>.ADMN@smil.mil. From everything I read, it made sense to use PIV plus OIDs (I can see multiple OIDs in my cert)

[jencot01]

EDIPI will NOT work per account formatting in your last reply.  You will definitely need PIV.  

Have you tried to sign into Splunk via token using a non-admin account?

In the web.conf help page, it gives the different values you can use for certBasedUserAuthMethod.  PIV would be correct for you, but the certBasedUserAuthPivOidList may require a different value.  I would look at your CAC values and find the field/attribute that holds the value you need Splunk to read.  Per web.conf help page, https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Webconf

PIV (Personal Identity Verification): Use PIV, a 16-digit numeric identifier typically formatted 
    as xxxxxxxxxxxxxxxx@mil. It is extracted from an "Other Name" field in the Subject Alternate Name which 
    corresponds to one of the object identifiers (OIDs) that you configure in 'certBasedUserAuthPivOidList'.

Seems like the incorrect field is being read.  Look through your logs to see if it shows the value that is being read in and try to match that value up on your CAC.

Otherwise, here is the full configuration for web.conf CAC authentication that I've had success with:

[settings]

requireClientCert = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert_chain_created.pem
enableCertBasedUserAuth = true
SSOMode = permissive
trustedIP = 127.0.0.1
certBasedUserAuthMethod = PIV 
certBasedUserAuthPivOidList = Microsoft Universal Principal Name
allowSsoWithoutChangingServerConf = 1 

[jnoose]

Thanks for the quick reply! Which logs should I be parsing to find the value that is being read? Logs on the splunk server or the windows domain side? 

So far, I've just been tailing the splunkd.log 

[jencot01]

Should be in the splunkd.log.  Here is an example from someone's previous post:

09-29-2023 09:02:43.191 -0400 INFO  AuthenticationProviderLDAP [12404 TcpChannelThread] - Could not find user=" \x84\x07\xd8\xb6\x05" with strategy="123_LDAP"
09-29-2023 09:02:43.192 -0400 ERROR HTTPAuthManager [12404 TcpChannelThread] - SSO failed - User does not exist:  \x84\x07\xd8\xb6\x05

[jencot01]

If you are not seeing failed logins in your splunkd.log, you can try updating the log.cfg or log-local.cfg file to add debugging.  This should give you more information in the splunkd.log.  The log.cfg/log-local.cfg file is located in the .../splunk/etc directory.

Find "category.AuthenticationProviderLDAP=INFO" and change INFO to DEBUG.

Restart the Splunk service.

This should at least give you the username it is finding.  There may be other options you can change to DEBUG to give you more information.

[jnoose]

I configured the DEBUG logs and will hopefully have more to go off next week. One thing I thought I'd bring up is some confusion regarding the Certificate Profile, Subject Alternative Names (SANs) and Extended Key Usage we should be specifying when we send our CSRs to the high-side PKI portal. 

Certificate Profiles
We have a few certificate profiles to choose from, including: device, domain controller, device/TLS/Application Email, Mini Crypto Key Agreement, Encrypted File System, IPSEC, Mini Crypro Authentication, Robotic Process (RPA/BO), and what seemed to be the most fitting, TLS Server. 

Subject Alternative Names (loopback?)
From there, we've been defining the Subject Alternative Name which makes the most sense, the device IP dress. However, I'm being told we should be using 127.0.0.1 instead - what's your take on that?

Key Usage/Extended Key Usage Selections
When selecting the TLS Server certificate profile, the default Key Usages are selected: 

• Key Usage:

  • digitalSignature

  • keyEncipherment

• Extended Key Usage:

  • id-kp-serverAuth

I'm being told that we need to include Extended Keys for smartCardAuth and possibly  id-kp-clientAuth. The problem is that when we select additional keys for the TLS Server profile the portal, instead automatically approving the CSR, kicks it back with the following error: "An extended key usage was found that requires the certificate application to be queued. EKU for "smarCardLogon" for profile "tlsServer" requires the certificate application to be queued"  This MASSIVELY slows down the troubleshooting process, making it quite difficult to iteratively troubleshoot (not Splunk's problem, but I needed to lament the hardship).

I know this is a LOT and truly appreciate everyone's help on this. Some my peers have been trying to figure this our for over a year! If we can figure this out it'd be a massive win for whole bunch of burnt out sys admins ❤️ 

goal/tl:dr - confirm the minimal, PKI-approved certificate profile, SAN list, and EKU set that Splunk Web needs for CAC / SIPR-token authentication, so we can eliminate certificate-format variables from our troubleshooting.  Thanks again!

[jencot01]

I'm not sure if anyone has found the exact problem in your situation, but looks like you may missing the attribute certBasedUserAuthPivOidList.  I do see errors for OID not found in client cert.  The default value is Microsoft Universal Principal Name, but you may need to change it.  Or try changing certBasedUserAuthMethod from PIV to EDIPI.    Hope this helps.

[MattD]

Did you get this figured out? We are currently fighting the same issue.

[mdmorgan]

Were you able to find any resolution to this?