Monitoring Splunk

Missing Audit logs

Path Finder

I just noticed that our Redhat splunk servers are missing audit log data for users logging in to Splunk.

For example, this query no longer returns data:
index=_audit action="login attempt" "info=succeeded"

I do have some audit data, just not the login attempts.

The data seems to of stopped after upgrading to version >=8.0.0

I only have one windows splunk server, and ALL the audit data appears to be there.

Labels (1)
0 Karma

Path Finder

I engaged splunk support and there was a code change around version 8.x that might have caused this to stop working.

Once I have a workaround I will post it here.

0 Karma

Motivator

Sounds like your search heads are no longer forwarding internal logs to the indexer cluster.
Ensure they are configured to do so by examining $SPLUNK_HOME/etc/system/local/outputs.conf to verify the SHC is sending those logs to the indexers. And/or look at inputs.conf to verify there are no blacklists that might be blocking those logs.

https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/Outputsconf#outputs.conf.example

0 Karma

Path Finder

We double-checked the outputs and don't see any errors. The indexer is getting some events (e.g failed logins are showing up, just not the successful logins. )

I will take a look at the inputs again just to make sure there are no problems with our blacklisting.

What's strange is the successful events aren't actually on the local search heads logs
(/$SPLUNK_HOME/var/splunk/log/audit.log)

That's why I leaning towards a change in functionality with the 8.0 release.

0 Karma