Unable to read audit.log file

[premg]

Splunk by default monitors /opt/splunk/var/log/splunk folder in Splunk Universal Forwarder.
But I am not able to see "audit.log" file in Splunk Web.
I am able to see the file when I execute list monitor command.

Also I tried to monitor that file separately by putting a monitor statement in etc/apps folder.
But still I'm not able to see that file in the Splunk Web.

The monitor statements looks as below.

[monitor:///opt/splunkforwarder/var/log/splunk/audit.log]
disabled = false
index = test_index
sourcetype = test_audit_log

Could you please help me in getting the audit.log file?

[MuS]

Hi premg,

first make sure your internal logs are forwarded from the universal forwarder towards the indexer, this is only in Splunk 6 UF default for _audit. If so you can search like this for the events on the indexer:

index=_audit

cheers, MuS

[jplumsdaine22]

@Mus - do the UF's actually generate audit events?

@premg - is there actually data on the forwarder in /opt/splunk/var/log/splunk/audit.log ? I strongly suspect there is nothing there to actually be monitored

[vincenp2]

granted there's not much but there is data there - mainly from when splunk is stopped / started, and when conf files have been modified

[jplumsdaine22]

can you post the relevant monitor stanza from $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug ?

[vincenp2]

apologies for the delay in responding -

The btool output from servers NOT reporting /opt/splunk/var/log/splunk/audit.log events (SPLUNKFORWARDER v6.2.5 deployed) - show the following stanzas exist (note that metrics.log and splunkd.log ARE being reported):

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:////opt/splunkforwarder/var/log/splunk/splunkd.log]
/opt/splunkforwarder/etc/system/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]

I've also checked the output from the btool command on a server which IS reporting/opt/splunk/var/log/splunk/audit.log events (as well as metrics.log and splunkd.log) and the only stanza which relates is as below, but as this is in a default directory I am assuming (perhaps incorrectly) that this has no impact? note SPLUNK v6.2.5 deployed

/opt/splunk/etc/system/default/inputs.conf [monitor:///opt/splunk/var/log/splunk]

ALL the stanzas above relate to files in 'default' directories, obviously correct me if I'm wrong but these shouldn't have any impact whatsoever should they?

[MuS]

So just checked on a Splunk universal forwarder 6.4.0 on Linux and there is an audit.log in /opt/splunkforwarder/var/log/splunk/ and it contains useful information. For example:

04-27-2016 08:30:40.226 +1200 INFO  AuditLogger - Audit:[timestamp=04-27-2016 08:30:40.226, user=n/a, action=update,path="/opt/splunkforwarder/etc/apps/splunk_TA_nix_local_log/bin", isdir=1, size=4096, gid=1001, uid=1001, modtime="Wed Apr 27 08:28:17 2016", mode="rwxrwxr-x", hash=, chgs="modtime "][n/a]
04-27-2016 08:30:40.330 +1200 INFO  AuditLogger - Audit:[timestamp=04-27-2016 08:30:40.330, user=n/a, action=update,path="/opt/splunkforwarder/etc/apps/splunk_TA_nix_local_log/bin/tests.sh", isdir=0, size=336, gid=1001, uid=1001, modtime="Wed Apr 27 08:28:17 2016", mode="rwxrwxr-x", hash=, chgs="modtime "][n/a]
04-27-2016 10:31:45.225 +1200 INFO  AuditLogger - Audit:[timestamp=04-27-2016 10:31:45.225, user=n/a, action=splunkShuttingDown, info=n/a][n/a]
04-27-2016 10:31:49.783 +1200 INFO  AuditLogger - Audit:[timestamp=04-27-2016 10:31:49.783, user=n/a, action=splunkStarting, info=n/a][n/a]

But audit.log is not added as monitor:

/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf            [monitor:///Library/Logs]
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf            [monitor:///etc]
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf            [monitor:///home/.../.bash_history]
/opt/splunkforwarder/etc/system/default/inputs.conf                        [monitor:///opt/splunkforwarder/etc/splunk.version]
/opt/splunkforwarder/etc/system/default/inputs.conf                        [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf            [monitor:///root/.bash_history]

So maybe this was changed somewhen down the road or it's a feature 😉

[premg]

Thanks MuS.

I have also tried with index=_audit. But no luck.

Also I am able to see /opt/splunk/var/log/splunk/audit.log path in the list monitor. So I believe it is monitored.

But not able to search.

Do I need to change anything else?

[jplumsdaine22]

Can you post the search you are using?

[vincenp2]

thanks for replying - the search I am using is
index=_* host=*
this returns some hosts producing audittrail events, these are splunk indexers and heavy forwarders using the full splunk deployment.

All servers which have splunkforwarder deployed and reporting to the heavy forwarder produce events from metrics.log and splunkd.log, but not audit.log

I hope this information helps - if you need more then please let me know

[vincenp2]

I seem to be failing miserably typing in the search I am using - here it is in words - hope it makes sense
index equals underscore asterisk host equals asterisk

[jplumsdaine22]

Yeah its not that intuitive how to put code samples in here.

Enter a new line and indent 4 spaces

like this

[vincenp2]

index=_* host=*

[vincenp2]

I am experiencing this same problem - I can see logfiles /opt/splunkforwarder/var/log/metrics.log and also splunkd.log being monitored - but not audit.log - can anyone suggest where to look for a solution please?