Splunk CAC Based Authentication

[partom24]

Hello All!

Trying to set up CAC Based Auth for SPLUNK 9.1.1 on Windows Server 2022 for the first time. I have successfully setup LDAP and am able to sign into Splunk using an AD username/password without any issues. When I add in the requiredClientCert, enableCertBasedAuth and certBasedUserAuthMethod stanzas, and attempt to access the Splunk GUI, all users are immediately greeted with an 'Unauthorized' message. I've been fighting this for about a week now, and Splunk support hasn't been able to help me pin this down yet. Any assistance would be greatly appreciated.

I've ensured TLS 1.2 registry keys exist in SCHANNEL to Enable TLS 1.2.

Corresponding logs from splunkd.log for the logon attempt are:

09-29-2023 09:02:43.191 -0400 INFO  AuthenticationProviderLDAP [12404 TcpChannelThread] - Could not find user=" \x84\x07\xd8\xb6\x05" with strategy="123_LDAP"
09-29-2023 09:02:43.192 -0400 ERROR HTTPAuthManager [12404 TcpChannelThread] - SSO failed - User does not exist:  \x84\x07\xd8\xb6\x05
09-29-2023 09:02:43.192 -0400 ERROR UiAuth [12404 TcpChannelThread] - user= \x84\x07\xd8\xb6\x05 action=login status=failure reason=sso-failed useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" clientip=<ip>
09-29-2023 09:03:10.247 -0400 ERROR UiAuth [12404 TcpChannelThread] - SAN OtherName not found for configured OIDs in client certificate
09-29-2023 09:03:10.247 -0400 ERROR UiAuth [12404 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

authentication.conf:

[splunk_auth]
minPasswordLength = 8
minPasswordUppercase = 0
minPasswordLowercase = 0
minPasswordSpecial = 0
minPasswordDigit = 0

[authentication]
authSettings = 123_LDAP
authType = LDAP

[123_LDAP]
SSLEnabled = 1
anonymous_referrals = 0
bindDN = CN=<Account>,OU=Service Accounts,OU=<Command Accounts>,DC=<Command>,DC=NAVY,DC=MIL
bindDNpassword = <removed>
charset = utf8
emailAttribute = mail
enableRangeRetrieval = 0
groupBaseDN = OU=SPLUNK Groups,OU=Groups,DC=<command>,DC=NAVY,DC=MIL
groupMappingAttribute = dn
groupMemberAttribute = member
groupNameAttribute = cn
host = DC.<Command>.NAVY.MIL
nestedGroups = 1
network_timeout = 20
pagelimit = -1
port = 636
realNameAttribute = displayName
sizelimit = 1000
timelimit = 15
userBaseDN = OU=Users,OU=<Command Accounts>,DC=<Command>,DC=NAVY,DC=MIL
userNameAttribute = userprincipalname

[roleMap_LDAP]
admin = SPLUNK AUDITOR
can_delete = SPLUNK AUDITOR
network = SPLUNK NETWORK
user = SPLUNK AUDITOR;SPLUNK USERS

web.conf

[settings]
enableSplunkWebSSL = true
privKeyPath = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_key.pem
serverCert = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_server.pem
sslPassword = <removed>
requireClientCert = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\dodCerts\DoDRootCA3.pem
enableCertBasedUserAuth=true
SSOMode=permissive
trustedIP = 127.0.0.1
certBasedUserAuthMethod=PIV

server.conf

[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\dodCerts\DoDRootCA3.pem
serverCert = $SPLUNK_HOME\etc\auth\dodCerts\splunk2_server.pem
sslPassword = <removed>
cliVerifyServerName = true
sslVersions = tls1.2
sslVerifyServerCert = true

[general]
serverName = SPKVSPLUNK2
pass4SymmKey = <removed>
trustedIP = 127.0.0.1

[jnoose]

Here are the confs that worked for us: 

server.conf

[general]
serverName = [splunkhostname]
pass4SymmKey = [pass4SymmKey]
sessionTimeout = 15m

[sslConfig]
sslPassword = [sslPassword]
sslRootCAPath = /opt/splunk/etc/auth/dod_chain.pem
sslPassword = [pw-hash]


### Omitting lmppol, license, kvstore, diskusage setttings ###

 
web.conf

[settings]
### START SPLUNK WEB USING HTTPS:8443 ###
enableSplunkWebSSL = 1
httpport = 8443
privKeyPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\privkey.pem
serverCert =  $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert.pem   

### TOKEN AUTHENTICATION ###
requireClientCert = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\dod_chain.pem
enableCertBasedUserAuth = true
SSOMode = permissive
trustedIP = 127.0.0.1
certBasedUserAuthMethod = PIV 
certBasedUserAuthPivOidList = Microsoft Universal Principal Name
allowSsoWithoutChangingServerConf = 1    

### Omitting STIG Settings (e.g., session timeout, login banner, etc). 

authentication.conf

### [Omitting splunk_auth password/user policies]

[authentication]
authSettings = ISXX DC-01 LDAPS Authentication, ISXX LDAPS Authentication
authType = LDAP

[roleMap_ISXX DC-01 LDAPS Authentication]
admin = Network Administrators
power = Network Administrators
user = Domain Admins; Network Administrators;Protected Users

[roleMap_ISXX DC-02 LDAPS Authentication]
admin = Network Administrators
power = Network Administrators
user = Domain Admins; Network Administrators;Protected Users

[ISXX DC-01 LDAPS Authentication]
SSLEnabled = 1
anonymous_referrals = 0
bindDN = CN=ldap.splunk,OU=Privileged Users,DC=XXXX,DC=YYYY
bindDNpassword = [pw-hash]
charset = utf8
emailAttribute = mail
enableRangeRetrieval = 0
groupBaseDN = CN=Network Administrators,OU=Users,DC=XXXX,DC=YYYY
groupMappingAttribute = dn
groupMemberAttribute = member
groupNameAttribute = cn
host = dc-01.XXXX.YYYY
nestedGroups = 0
network_timeout = 29
pagelimit = -1
port = 636
realNameAttribute = cn
sizelimit = 5000
timelimit = 25
userBaseDN = OU=Privileged Users,DC=XXXX,DC=YYYY
userNameAttribute = userPrincipalName

[ISXX DC-02 LDAPS Authentication]
SSLEnabled = 1
anonymous_referrals = 0
bindDN = CN=ldap.splunk,OU=Privileged Users,DC=XXXX,DC=YYYY
bindDNpassword = [pw-hash]
charset = utf8
emailAttribute = mail
enableRangeRetrieval = 0
groupBaseDN = CN=Network Administrators,OU=Users,DC=XXXX,DC=YYYY
groupMappingAttribute = dn
groupMemberAttribute = member
groupNameAttribute = cn
host = dc-01.XXXX.YYYY
nestedGroups = 0
network_timeout = 29
pagelimit = -1
port = 636
realNameAttribute = cn
sizelimit = 5000
timelimit = 25
userBaseDN = OU=Privileged Users,DC=XXXX,DC=YYYY
userNameAttribute = userPrincipalName

[skurasak1]

I believe I have things setup similarly but after I get CAC PIN pop up, It just goes to the standard login page.  I can login with my ldap setup and can login with PIV@mil at the login prompt, but shouldn't thing bypass the login once the CAC/PIN is successful?  What am I missing?

[jencot01]

I have run into that issue before in configuring CAC/Token login.  I can't remember if this was the reason, but make sure in the authentication.conf the host field for LDAP server needs to be the FQDN of the server and not the IP address.

[skurasak1]

Thanks, I did have the FQDN, so the search is still on, hopefully others had the similar issue and resolution.  Any ideas on which log would show why, I'm not able to find one that would give me a hint.

[jnoose]

Ladies and Gentlemen, we got it working 🎉 Some quick lessons learned:

1. It turns out you do not need the Extend Key Usage for smartCardLogon when you submit your CSR to the DoD NPE portal. A simple TLS Server request with the defaults will work. 
2. httpport must be 8443. 
3. certBasedAuthMethod = Microsoft Universal Principal Name (NOT PIV - which also removes the need for the certBasedUserAuthPivOidList attribute)

With that, all we did was follow this tutorial:
https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Configuring_Splunk_for_Common...

[jnoose]

MY BAD on #3, meant to write:

3. certBasedAuthMethod = Microsoft Universal Principal Name (NOT a specific OID )

Also, here's the how-to for the NPE portal. 

There has been a lot of confusion on which certificate type, the long list CC/S/A's, and key selections. Although the CC/S/A's might vary across circuits (Westford is using DOD), the common criteria is:

Authenticate to the NPE Portal with you .DA token for instant approval

Email: Your SIPR email (I don't think this matters)

Subject/CN: device DNS name (automatically appears after pasting the CSR text)

Certificate Profile: TLS Server

Key Usage Selections: digitalSignature, keyEnchipherment 

Extended Key Usage Options: id-kp-serverAuth

Subject Alternative Name: + Device DNS Hostname w/FQDN, + IP Address (the actual IP, not 127.0.0.1 as was discussed in some channels)

CC/S/A: DOD (yours may be different)

Validity: Will default to 1-year, manually increase to 3.

[SPL_Dummy]

Following for I need to do this soon as well... hope you figure it out so I can 😉

[jnoose]

Anyone, anyone? Bueller?

I feel like I'm so close to making this work as well. SSL/TLS is configured, Splunk Web GUI prompts for PIV token + PIN, but it fails out to some "ERROR: Unauthorized" xml garbage in the browser. 

Tailing splunkd for CertBasedUserAuth reveals: error fetching username from client certificate. 

Relevant Settings: 
certBasedUserAuthMethod = PIV

certBasedUserPivOidList = 1.3.6.1.4.1.311.20.2.3,Microsoft Universal Principal Name

Any ideas? 

[jencot01]

Have you tried replacing PIV with EDIPI?  

certBasedUserAuthMethod = EDIPI

[jnoose]

No dice.

Previous errors with PIV/OID were 

ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate

ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

New error with EDIPI

ERROR UiAuth [2488903 TcpChannelThread] - user=<DoDID#> action=login status=failure reason=sso-failed useragent=<browser stuff>

[jencot01]

I just noticed one of your last posts showing the following errors:

Previous errors with PIV/OID were 

ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate

ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate

The PIV pulls from the Subject Alternate Name (SAN) "Other Name."  Validate on your PIV the value of Other Name in Subject Alternate Name.  I'm assuming the value you would like to pull is not found in that location.  You will need to find the OID value for the location on your PIV and change the certBasedUserAuthPivOidList values to match the correct location on your PIV. 

Hope this helps.

[jnoose]

Great catch. I noticed this as well and thought I had a smoking gun in 90Meter Smart Card Manager for my token. I noticed: 

Extension Type: Subject Alternative Name
       Oid: 2.5.29.17
              Other Name: Principal Name=<DoD-ID>.ADMN@smil.mil

I put that Oid into the web.conf, restarted, and got the same UiAuth erros 😞

[jencot01]

If you are having the same issue next week, I will be in an environment that I can help better.  I'm currently going off of memory.  Please let me know if you still need help on Monday and I can help troubleshoot better.  Sorry I can't think of anything else to suggest right now.

[jnoose]

Hey there! Any chance you would still be able to help with this?

[jnoose]

I'm going to try and push through today but I reckon I'll be in the same place Monday. I really appreciate your help so far and would be so grateful if dig in deeper next week. Thanks again and enjoy your weekend!

[jnoose]

I'll give it go. PIV just seems like the way to go because my UPN is <myDoDID#>.ADMN@smil.mil. From everything I read, it made sense to use PIV plus OIDs (I can see multiple OIDs in my cert)

[jencot01]

EDIPI will NOT work per account formatting in your last reply.  You will definitely need PIV.  

Have you tried to sign into Splunk via token using a non-admin account?

In the web.conf help page, it gives the different values you can use for certBasedUserAuthMethod.  PIV would be correct for you, but the certBasedUserAuthPivOidList may require a different value.  I would look at your CAC values and find the field/attribute that holds the value you need Splunk to read.  Per web.conf help page, https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Webconf

PIV (Personal Identity Verification): Use PIV, a 16-digit numeric identifier typically formatted 
    as xxxxxxxxxxxxxxxx@mil. It is extracted from an "Other Name" field in the Subject Alternate Name which 
    corresponds to one of the object identifiers (OIDs) that you configure in 'certBasedUserAuthPivOidList'.

Seems like the incorrect field is being read.  Look through your logs to see if it shows the value that is being read in and try to match that value up on your CAC.

Otherwise, here is the full configuration for web.conf CAC authentication that I've had success with:

[settings]

requireClientCert = true
sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert_chain_created.pem
enableCertBasedUserAuth = true
SSOMode = permissive
trustedIP = 127.0.0.1
certBasedUserAuthMethod = PIV 
certBasedUserAuthPivOidList = Microsoft Universal Principal Name
allowSsoWithoutChangingServerConf = 1 

[jnoose]

Thanks for the quick reply! Which logs should I be parsing to find the value that is being read? Logs on the splunk server or the windows domain side? 

So far, I've just been tailing the splunkd.log 

[jencot01]

Should be in the splunkd.log.  Here is an example from someone's previous post:

09-29-2023 09:02:43.191 -0400 INFO  AuthenticationProviderLDAP [12404 TcpChannelThread] - Could not find user=" \x84\x07\xd8\xb6\x05" with strategy="123_LDAP"
09-29-2023 09:02:43.192 -0400 ERROR HTTPAuthManager [12404 TcpChannelThread] - SSO failed - User does not exist:  \x84\x07\xd8\xb6\x05

[jencot01]

If you are not seeing failed logins in your splunkd.log, you can try updating the log.cfg or log-local.cfg file to add debugging.  This should give you more information in the splunkd.log.  The log.cfg/log-local.cfg file is located in the .../splunk/etc directory.

Find "category.AuthenticationProviderLDAP=INFO" and change INFO to DEBUG.

Restart the Splunk service.

This should at least give you the username it is finding.  There may be other options you can change to DEBUG to give you more information.