Mine is  -----BEGIN PRIVATE KEY----- gobbleygook -----END PRIVATE KEY----- ... View more

These are the instructions we followed: https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Configuring_Splunk_for_Common_Access_Card_(CAC)_authentication I recall this command not working:    I don't know HOW this worked but we just ran "mv mySplunkWebPrivateKey.key privkey.pem" ... ... View more

Are you using port 8443??? I don't see the paths to your privkey/server certs.   [settings] ### START SPLUNK WEB USING HTTPS:8443 ### enableSplunkWebSSL = 1 httpport = 8443 privKeyPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\privkey.pem serverCert = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert.pem ### TOKEN AUTHENTICATION ### requireClientCert = true sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\dod_chain.pem enableCertBasedUserAuth = true SSOMode = permissive trustedIP = 127.0.0.1 certBasedUserAuthMethod = PIV certBasedUserAuthPivOidList = Microsoft Universal Principal Name allowSsoWithoutChangingServerConf = 1 ... View more

Correct - all I requested from DoD NPE was a vanilla 3-year TLS cert (no extended key usage). Yes, we have this working on 9.x on [X]IPR. Might there be something wrong with your SmarCard middleware? Not being able to decrypt the token sounds like maybe there's an issue passing the PIN?  ... View more

How did you create the certs? DoD NPE Portal?  We got it working with just "certBasedUserAuthPivOidList =  Microsoft Universal Principal Name".  I shared my configs further up, please have a look ... View more

Here are the confs that worked for us:  server.conf [general] serverName = [splunkhostname] pass4SymmKey = [pass4SymmKey] sessionTimeout = 15m [sslConfig] sslPassword = [sslPassword] sslRootCAPath = /opt/splunk/etc/auth/dod_chain.pem sslPassword = [pw-hash] ### Omitting lmppol, license, kvstore, diskusage setttings ###   web.conf [settings] ### START SPLUNK WEB USING HTTPS:8443 ### enableSplunkWebSSL = 1 httpport = 8443 privKeyPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\privkey.pem serverCert = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert.pem ### TOKEN AUTHENTICATION ### requireClientCert = true sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\dod_chain.pem enableCertBasedUserAuth = true SSOMode = permissive trustedIP = 127.0.0.1 certBasedUserAuthMethod = PIV certBasedUserAuthPivOidList = Microsoft Universal Principal Name allowSsoWithoutChangingServerConf = 1 ### Omitting STIG Settings (e.g., session timeout, login banner, etc).   authentication.conf ### [Omitting splunk_auth password/user policies] [authentication] authSettings = ISXX DC-01 LDAPS Authentication, ISXX LDAPS Authentication authType = LDAP [roleMap_ISXX DC-01 LDAPS Authentication] admin = Network Administrators power = Network Administrators user = Domain Admins; Network Administrators;Protected Users [roleMap_ISXX DC-02 LDAPS Authentication] admin = Network Administrators power = Network Administrators user = Domain Admins; Network Administrators;Protected Users [ISXX DC-01 LDAPS Authentication] SSLEnabled = 1 anonymous_referrals = 0 bindDN = CN=ldap.splunk,OU=Privileged Users,DC=XXXX,DC=YYYY bindDNpassword = [pw-hash] charset = utf8 emailAttribute = mail enableRangeRetrieval = 0 groupBaseDN = CN=Network Administrators,OU=Users,DC=XXXX,DC=YYYY groupMappingAttribute = dn groupMemberAttribute = member groupNameAttribute = cn host = dc-01.XXXX.YYYY nestedGroups = 0 network_timeout = 29 pagelimit = -1 port = 636 realNameAttribute = cn sizelimit = 5000 timelimit = 25 userBaseDN = OU=Privileged Users,DC=XXXX,DC=YYYY userNameAttribute = userPrincipalName [ISXX DC-02 LDAPS Authentication] SSLEnabled = 1 anonymous_referrals = 0 bindDN = CN=ldap.splunk,OU=Privileged Users,DC=XXXX,DC=YYYY bindDNpassword = [pw-hash] charset = utf8 emailAttribute = mail enableRangeRetrieval = 0 groupBaseDN = CN=Network Administrators,OU=Users,DC=XXXX,DC=YYYY groupMappingAttribute = dn groupMemberAttribute = member groupNameAttribute = cn host = dc-01.XXXX.YYYY nestedGroups = 0 network_timeout = 29 pagelimit = -1 port = 636 realNameAttribute = cn sizelimit = 5000 timelimit = 25 userBaseDN = OU=Privileged Users,DC=XXXX,DC=YYYY userNameAttribute = userPrincipalName ... View more

MY BAD on #3, meant to write: 3. certBasedAuthMethod = Microsoft Universal Principal Name (NOT a specific OID ) Also, here's the how-to for the NPE portal.  There has been a lot of confusion on which certificate type, the long list CC/S/A's, and key selections. Although the CC/S/A's might vary across circuits (Westford is using DOD), the common criteria is:   Authenticate to the NPE Portal with you .DA token for instant approval   Email: Your SIPR email (I don't think this matters) Subject/CN: device DNS name (automatically appears after pasting the CSR text) Certificate Profile: TLS Server Key Usage Selections: digitalSignature, keyEnchipherment  Extended Key Usage Options: id-kp-serverAuth Subject Alternative Name: + Device DNS Hostname w/FQDN, + IP Address (the actual IP, not 127.0.0.1 as was discussed in some channels) CC/S/A: DOD (yours may be different) Validity: Will default to 1-year, manually increase to 3. ... View more

Ladies and Gentlemen, we got it working 🎉 Some quick lessons learned: It turns out you do not need the Extend Key Usage for smartCardLogon when you submit your CSR to the DoD NPE portal. A simple TLS Server request with the defaults will work.  httpport must be 8443.  certBasedAuthMethod = Microsoft Universal Principal Name (no need to manually define OIDs) With that, all we did was follow this tutorial: https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Configuring_Splunk_for_Common_Access_Card_(CAC)_authentication   ... View more

Hey there! Any chance you would still be able to help with this? ... View more

I forgot what it felt like for someone to copy my assignment in homeroom 😂 ... View more

I encountered nearly identical symptoms on one of my RHEL9 systems. The key errors you mentioned: "splunkd is not running. failed splunkd.pid doesn't exist" "tcp_conn_open_afux ossocket_connect failed with no such file or directory" Forwarder showing as "inactive" despite correct configuration These are classic signs of what I discovered was a "split brain" situation with duplicate systemd service files. While your reinstall fixed it (likely by cleaning up these duplicate files), others might benefit from understanding the root cause: Check for duplicate service files: ls -la /usr/lib/systemd/system/SplunkForwarder.servicels -la /etc/systemd/system/SplunkForwarder.service If both exist, that's the problem! The one in /etc/systemd/system/ takes precedence and might have different user/permissions settings. In my case, one was set to run as SplunkFwd user while the other was running as root. This causes a situation where: Systemd shows SplunkForwarder running Splunk CLI thinks it's not running Permission conflicts prevent proper operation PID file issues occur Connection failures happen despite proper network connectivity The fix is simpler than reinstalling: sudo rm /etc/systemd/system/SplunkForwarder.servicesudo systemctl daemon-reloadsudo systemctl restart SplunkForwarder This can happen when multiple installation methods are used (like RPM install + splunk enable boot-start). Sharing this because my "nuke and pave" initially didn't work either until I discovered this specific issue.  Being in the DoD air-gap hell made this even harder to troubleshoot! ... View more

I configured the DEBUG logs and will hopefully have more to go off next week. One thing I thought I'd bring up is some confusion regarding the Certificate Profile, Subject Alternative Names (SANs) and Extended Key Usage we should be specifying when we send our CSRs to the high-side PKI portal.  Certificate Profiles We have a few certificate profiles to choose from, including: device, domain controller, device/TLS/Application Email, Mini Crypto Key Agreement, Encrypted File System, IPSEC, Mini Crypro Authentication, Robotic Process (RPA/BO), and what seemed to be the most fitting, TLS Server.  Subject Alternative Names (loopback?) From there, we've been defining the Subject Alternative Name which makes the most sense, the device IP dress. However, I'm being told we should be using 127.0.0.1 instead - what's your take on that? Key Usage/Extended Key Usage Selections When selecting the TLS Server certificate profile, the default Key Usages are selected:  Key Usage: digitalSignature keyEncipherment Extended Key Usage: id-kp-serverAuth I'm being told that we need to include Extended Keys for smartCardAuth and possibly  id-kp-clientAuth. The problem is that when we select additional keys for the TLS Server profile the portal, instead automatically approving the CSR, kicks it back with the following error: "An extended key usage was found that requires the certificate application to be queued. EKU for "smarCardLogon" for profile "tlsServer" requires the certificate application to be queued"  This MASSIVELY slows down the troubleshooting process, making it quite difficult to iteratively troubleshoot (not Splunk's problem, but I needed to lament the hardship). I know this is a LOT and truly appreciate everyone's help on this. Some my peers have been trying to figure this our for over a year! If we can figure this out it'd be a massive win for whole bunch of burnt out sys admins ❤️    goal/tl:dr - confirm the minimal, PKI-approved certificate profile, SAN list, and EKU set that Splunk Web needs for CAC / SIPR-token authentication, so we can eliminate certificate-format variables from our troubleshooting.  Thanks again! ... View more

I'm going to try and push through today but I reckon I'll be in the same place Monday. I really appreciate your help so far and would be so grateful if dig in deeper next week. Thanks again and enjoy your weekend! ... View more

Great catch. I noticed this as well and thought I had a smoking gun in 90Meter Smart Card Manager for my token. I noticed:  Extension Type: Subject Alternative Name        Oid: 2.5.29.17               Other Name: Principal Name=<DoD-ID>.ADMN@smil.mil I put that Oid into the web.conf, restarted, and got the same UiAuth erros 😞 ... View more

Thanks for the quick reply! Which logs should I be parsing to find the value that is being read? Logs on the splunk server or the windows domain side?  So far, I've just been tailing the splunkd.log    ... View more

No dice. Previous errors with PIV/OID were  ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate   New error with EDIPI ERROR UiAuth [2488903 TcpChannelThread] - user=<DoDID#> action=login status=failure reason=sso-failed useragent=<browser stuff> ... View more

I'll give it go. PIV just seems like the way to go because my UPN is <myDoDID#>.ADMN@smil.mil. From everything I read, it made sense to use PIV plus OIDs (I can see multiple OIDs in my cert) ... View more

Anyone, anyone? Bueller? I feel like I'm so close to making this work as well. SSL/TLS is configured, Splunk Web GUI prompts for PIV token + PIN, but it fails out to some "ERROR: Unauthorized" xml garbage in the browser.  Tailing splunkd for CertBasedUserAuth reveals: error fetching username from client certificate.  Relevant Settings:  certBasedUserAuthMethod = PIV certBasedUserPivOidList = 1.3.6.1.4.1.311.20.2.3,Microsoft Universal Principal Name   Any ideas?  ... View more

♪♫♬ And they say that a hero could saaaaaaave us I'm not gonna stand here and waaaaaaaait ♪♫♬ ... View more