I have run into that issue before in configuring CAC/Token login.  I can't remember if this was the reason, but make sure in the authentication.conf the host field for LDAP server needs to be the FQDN of the server and not the IP address. ... View more

If you are not seeing failed logins in your splunkd.log, you can try updating the log.cfg or log-local.cfg file to add debugging.  This should give you more information in the splunkd.log.  The log.cfg/log-local.cfg file is located in the .../splunk/etc directory.   Find "category.AuthenticationProviderLDAP=INFO" and change INFO to DEBUG. Restart the Splunk service. This should at least give you the username it is finding.  There may be other options you can change to DEBUG to give you more information. ... View more

If you are having the same issue next week, I will be in an environment that I can help better.  I'm currently going off of memory.  Please let me know if you still need help on Monday and I can help troubleshoot better.  Sorry I can't think of anything else to suggest right now. ... View more

I just noticed one of your last posts showing the following errors: Previous errors with PIV/OID were  ERROR UiAuth [2487972 TcpChannelThread] -  SAN OtherName not found for configured OIDs in client certificate ERROR UiAuth [2487972 TcpChannelThread] - CertBasedUserAuth: error fetching username from client certificate The PIV pulls from the Subject Alternate Name (SAN) "Other Name."  Validate on your PIV the value of Other Name in Subject Alternate Name.  I'm assuming the value you would like to pull is not found in that location.  You will need to find the OID value for the location on your PIV and change the certBasedUserAuthPivOidList values to match the correct location on your PIV.  Hope this helps. ... View more

Should be in the splunkd.log.  Here is an example from someone's previous post: 09-29-2023 09:02:43.191 -0400 INFO AuthenticationProviderLDAP [12404 TcpChannelThread] - Could not find user=" \x84\x07\xd8\xb6\x05" with strategy="123_LDAP" 09-29-2023 09:02:43.192 -0400 ERROR HTTPAuthManager [12404 TcpChannelThread] - SSO failed - User does not exist: \x84\x07\xd8\xb6\x05   ... View more

EDIPI will NOT work per account formatting in your last reply.  You will definitely need PIV.   Have you tried to sign into Splunk via token using a non-admin account? In the web.conf help page, it gives the different values you can use for certBasedUserAuthMethod.  PIV would be correct for you, but the certBasedUserAuthPivOidList may require a different value.  I would look at your CAC values and find the field/attribute that holds the value you need Splunk to read.  Per web.conf help page, https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Webconf PIV (Personal Identity Verification): Use PIV, a 16-digit numeric identifier typically formatted as xxxxxxxxxxxxxxxx@mil. It is extracted from an "Other Name" field in the Subject Alternate Name which corresponds to one of the object identifiers (OIDs) that you configure in 'certBasedUserAuthPivOidList'. Seems like the incorrect field is being read.  Look through your logs to see if it shows the value that is being read in and try to match that value up on your CAC. Otherwise, here is the full configuration for web.conf CAC authentication that I've had success with: [settings] requireClientCert = true sslRootCAPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert_chain_created.pem enableCertBasedUserAuth = true SSOMode = permissive trustedIP = 127.0.0.1 certBasedUserAuthMethod = PIV certBasedUserAuthPivOidList = Microsoft Universal Principal Name allowSsoWithoutChangingServerConf = 1   ... View more

Have you tried replacing PIV with EDIPI?   certBasedUserAuthMethod = EDIPI   ... View more

I'm not sure if anyone has found the exact problem in your situation, but looks like you may missing the attribute certBasedUserAuthPivOidList.  I do see errors for OID not found in client cert.  The default value is Microsoft Universal Principal Name, but you may need to change it.  Or try changing certBasedUserAuthMethod from PIV to EDIPI.    Hope this helps.   ... View more

I have a follow-up question on this explanation... If you have an index with this configuration: [index] homePath = volume:primary/index/db coldPath = volume:primary/index/colddb thawedPath = $SPLUNK_DB/index/thaweddb tstatsHomePath = volume:primary/index/datamodel_summary maxTotalDataSizeMB = 102400 frozenTimePeriodInSecs = 31536000 (one year) coldToFrozenDir = /splunkdata/frozen/$_index_name If the maxTotalDataSizeMB is reached before frozenTimePeriodInSecs, does the data get deleted without archiving first or does it get archived first since coldToFrozenDir is configured? ... View more

To add to the above, I have also found this error in splunkd.log on the indexers: ERROR SearchProcessRunner  PreforkedSearchesManager-0   - preforked process = 0/253717  hung up This error aligns with the "broken pipe" error on my post above.  There are no skipped searches. I'm at a loss to what I steps I should take next to troubleshoot this issue.   Again, any help/suggestions would be appreciated. ... View more