Monitoring Splunk

Why are we Missing Audit logs?

RDAVISS
Path Finder

I just noticed that our Redhat splunk servers are missing audit log data for users logging in to Splunk.

For example, this query no longer returns data:

index=_audit action="login attempt" "info=succeeded"

I do have some audit data, just not the login attempts.

The data seems to of stopped after upgrading to version >=8.0.0

I only have one windows splunk server, and ALL the audit data appears to be there.

Labels (1)
0 Karma

dfronck
Communicator

@RDAVISS That search doesn't work if you have the Splunk_SA_CIM installed because "action" will never equal "login attempt"

[audittrail]
EVAL-action = case(match(_raw,"action\=login\sattempt") AND match(_raw,"info\=succeeded"),"success",match(_raw,"action\=login\sattempt") AND match(_raw,"info\=failed"),"failure",match(_raw,"action\=add"),"created",match(_raw,"action\=delete"),"deleted",match(_raw,"action\=update"),"modified",1=1,action)
EVAL-app = if(match(_raw,"action\=login\sattempt"),"splunk",app)


Try it without action=

index=_audit "login attempt" "info=succeeded"

 

0 Karma

RDAVISS
Path Finder

I engaged splunk support and there was a code change around version 8.x that might have caused this to stop working.

Once I have a workaround I will post it here.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Sounds like your search heads are no longer forwarding internal logs to the indexer cluster.
Ensure they are configured to do so by examining $SPLUNK_HOME/etc/system/local/outputs.conf to verify the SHC is sending those logs to the indexers. And/or look at inputs.conf to verify there are no blacklists that might be blocking those logs.

https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/Outputsconf#outputs.conf.example

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

RDAVISS
Path Finder

We double-checked the outputs and don't see any errors. The indexer is getting some events (e.g failed logins are showing up, just not the successful logins. )

I will take a look at the inputs again just to make sure there are no problems with our blacklisting.

What's strange is the successful events aren't actually on the local search heads logs
(/$SPLUNK_HOME/var/splunk/log/audit.log)

That's why I leaning towards a change in functionality with the 8.0 release.

CyberWarrior404
Observer

Did you manage to find the solution as to why the login activity is not showing up in Splunk 8.X?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

at least all in one node with 8.2.0 the event is still the same

Audit:[timestamp=05-29-2021 23:06:55.778, user=XXXXX, action=login attempt, info=succeeded reason=user-initiated useragent="Mozilla/5.0 (XXX; XXX ) XXXXX/....." clientip=127.0.0.1" method=Splunk" session=ecd9cadalsdklakdlakd8d5391718ea8]

I haven't access to distributed environment to check it (only 8.x.x).

index = _audit sourcetype=audittrail action="login attempt" info=*
| stats count by user,info

Previous example founds users as expected.

r. Ismo 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...