topic Re: Missing Audit logs in Monitoring Splunk

Why are we Missing Audit logs?

RDAVISS — Wed, 27 Apr 2022 15:38:27 GMT

I just noticed that our Redhat splunk servers are missing audit log data for users logging in to Splunk.

For example, this query no longer returns data:

index=_audit action="login attempt" "info=succeeded"

I do have some audit data, just not the login attempts.

The data seems to of stopped after upgrading to version >=8.0.0

I only have one windows splunk server, and ALL the audit data appears to be there.

Re: Missing Audit logs

codebuilder — Mon, 16 Mar 2020 22:49:43 GMT

Sounds like your search heads are no longer forwarding internal logs to the indexer cluster.
Ensure they are configured to do so by examining $SPLUNK_HOME/etc/system/local/outputs.conf to verify the SHC is sending those logs to the indexers. And/or look at inputs.conf to verify there are no blacklists that might be blocking those logs.

https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/Outputsconf#outputs.conf.example

Re: Missing Audit logs

RDAVISS — Thu, 26 Mar 2020 16:41:02 GMT

We double-checked the outputs and don't see any errors. The indexer is getting some events (e.g failed logins are showing up, just not the successful logins. )

I will take a look at the inputs again just to make sure there are no problems with our blacklisting.

What's strange is the successful events aren't actually on the local search heads logs
(/$SPLUNK_HOME/var/splunk/log/audit.log)

That's why I leaning towards a change in functionality with the 8.0 release.

Re: Missing Audit logs

RDAVISS — Fri, 03 Apr 2020 14:36:55 GMT

I engaged splunk support and there was a code change around version 8.x that might have caused this to stop working.

Once I have a workaround I will post it here.

Re: Missing Audit logs

CyberWarrior404 — Sat, 29 May 2021 06:53:04 GMT

Did you manage to find the solution as to why the login activity is not showing up in Splunk 8.X?

Re: Missing Audit logs

isoutamo — Sat, 29 May 2021 20:16:26 GMT

at least all in one node with 8.2.0 the event is still the same

Audit:[timestamp=05-29-2021 23:06:55.778, user=XXXXX, action=login attempt, info=succeeded reason=user-initiated useragent="Mozilla/5.0 (XXX; XXX ) XXXXX/....." clientip=127.0.0.1" method=Splunk" session=ecd9cadalsdklakdlakd8d5391718ea8]

I haven't access to distributed environment to check it (only 8.x.x).

index = _audit sourcetype=audittrail action="login attempt" info=* | stats count by user,info

Previous example founds users as expected.

r. Ismo

Re: Missing Audit logs

dfronck — Wed, 27 Apr 2022 13:53:34 GMT

@RDAVISS That search doesn't work if you have the Splunk_SA_CIM installed because "action" will never equal "login attempt"

[audittrail] EVAL-action = case(match(_raw,"action\=login\sattempt") AND match(_raw,"info\=succeeded"),"success",match(_raw,"action\=login\sattempt") AND match(_raw,"info\=failed"),"failure",match(_raw,"action\=add"),"created",match(_raw,"action\=delete"),"deleted",match(_raw,"action\=update"),"modified",1=1,action) EVAL-app = if(match(_raw,"action\=login\sattempt"),"splunk",app)

Try it without action=

index=_audit "login attempt" "info=succeeded"