Getting Data In

Why am I unable to monitor $SPLUNK_HOME/var/log/splunk/audit.log on my Linux Universal Forwarders?

Path Finder

I created an app named aufinputs_conf. The app simply contains inputs.conf that has the monitor stanza's below. This app was deployed to both Windows and Linux servers. It is working on the Windows servers, but not on the Linux servers.

[monitor://$SPLUNK_HOME\var\log\splunk\audit.log] 
_TCP_ROUTING = * 
index = _internal 

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log] 
_TCP_ROUTING = * 
index = _internal

I see log messages from splunkd.log that seem to indicate that Splunk is attempting to monitor the files, but when I search the _internal index for host=servername.domain and source="/opt/splunkforwarder/var/log/splunk/splunkd.log" i get no results.

02-14-2017 08:31:07.903 -0600 INFO  WatchedFile - Will begin reading at offset=111468 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 21:29:25.417 -0600 INFO  WatchedFile - Will begin reading at offset=108519 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 16:37:32.705 -0600 INFO  WatchedFile - Will begin reading at offset=105392 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 14:40:29.015 -0600 INFO  WatchedFile - Will begin reading at offset=103010 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 14:02:01.743 -0600 INFO  WatchedFile - Will begin reading at offset=100628 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 13:28:58.577 -0600 INFO  WatchedFile - Will begin reading at offset=96238 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd

I have verified that the app (inputs.conf file) is being deployed. The file permissions are okay. I'm not really sure what I should check next?

0 Karma
1 Solution

Path Finder

By default, Splunk audit events coming from non-Windows universal forwarders are sent to the null queue by way of system default configuration in props.conf.

#system/default/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit

As a workaround, we deployed an app with a local props.conf and referenced a non-existing transform to override the default of send_to _nullqueue .

#spp_cluster_indexer_base
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza 
sourcetype = splunk_audit

View solution in original post

Path Finder

By default, Splunk audit events coming from non-Windows universal forwarders are sent to the null queue by way of system default configuration in props.conf.

#system/default/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit

As a workaround, we deployed an app with a local props.conf and referenced a non-existing transform to override the default of send_to _nullqueue .

#spp_cluster_indexer_base
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza 
sourcetype = splunk_audit

View solution in original post

Communicator

Hi,

I tried with the posted solution but unfortunately it was not successful, therefore I worked directly with the Splunk support team to solve this, here below our findings.

Basically the audit.log from the Splunk Universal Forwarders is not indexed due to the fact that there is a default configuration that send it to null queue:

default/props.conf

 

[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit

 

even overwriting it with a local stanza was not working.

local/props.conf

 

[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza
sourcetype = splunk_audit

 

In order to solve it, it is needed to bypass the default configuration that is sending to null queue with a monitor stanza in inputs.conf, therefore I created a specific app in the Deployment Server and deployed to all the Splunk Universal Forwarders (*nix and Windows):

myapp/local/inputs.conf

 

# Specific configuration to enable monitoring Splunk Universal Forwarder audit logs
# by default they are sent to null queue

#*nix
[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = audittrail
source = audittrail

#Windows
[monitor://$SPLUNK_HOME\var\log\splunk\audit.log]
index = _audit
sourcetype = audittrail
source = audittrail

 

In this way the Splunk Universal Forwarders audit logs will be indexed in the same index and with the same source and sourcetype of the one coming from Splunk Enterprise servers (Search Heads, Indexers, Master Node, Deployment Server, Heavy Forwarders, License Master etc...).

NOTE:

sourcetype --> it need to be audittrail otherwise data are not ingested

source --> it need to be different from $SPLUNK_HOME/var/log/splunk/audit.log otherwise data are not ingested

I hope this can help you!

Best Regards,
Edoardo

0 Karma

Communicator

Here my answer on another thread to solve, once audit.log was ingested, the issue with merged events:

https://community.splunk.com/t5/Getting-Data-In/Splunk-Universal-Forwarders-audit-logs-merged-audit-...

0 Karma

Splunk Employee
Splunk Employee

Ah! After re-reading your question, the only thing I forgot to ask in my answer above was:

did you check index=_audit ?

On my nix system it throws my local audit logs in _audit, while my MS events are still in _internal

http://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Searchforauditevents

0 Karma

Path Finder

I did a search for the last 90 ( in our Production environment ) of index=_audit. All the results I find are from Linux servers running the Splunk server application. I don't find anything from a Linux server running the Splunk Universal Forwarder application.

I do think you are correct about audit.log being monitored by default, because I searched the Last 90 days (in a different environment) index=_internal sourcetype=splunk_audit and found 100's of Windows hosts logging messages from C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log.

The Production environment that I searched has 76 linux servers running the Universal Forwarder application. I found an /opt/splunkforwarder/var/log/splunk/audit.log/audit.log file with entries during that time frame.

0 Karma

Splunk Employee
Splunk Employee

yeah thats for sure the default and thats what I see for my windows host.

Looks like in my enviro the nix logs get flipped over to index=_audit sourcetype=audittrail which is a sourcetype that is in the CIM app. Do you see anything there? You are an admin user right?

0 Karma

Path Finder

I am in our Splunk admins AD group, but just to be certain I logged in as the user admin and searched index=_audit sourcetype=audittrail for the last 90 days.

The search returned 16,656,224 events, but the hosts were all splunk servers. Not anything from a host with the Universal Forwarder installed.

0 Karma

Splunk Employee
Splunk Employee

hmmm.....and I assume you have checked for index=* host=theforwarderyouarelookingfor source=*audit.log ?

Also have you checked locally on the UF to ensure there actually is any audit activity?

0 Karma

Path Finder

When I search index=* host=theforwarderiamlookingfor , I only get results from index=os , which is the index that our /var/log/secure goes to. Does searching index=* search _internal _audit, etc?

I found a Linux forwarder with log entries for audit.log in the last 90 days. I have a case open with support. If we get to the bottom of this, I will update this question with the resolution.

0 Karma

Splunk Employee
Splunk Employee

Hi matthewroberson69!

You actually shouldn't need to create a monitor for the audit log, as Splunk monitors it out of the box!

you can use btool to examine the inputs on your UF:

[splunker@n00b-splkufw-01 bin]$ ./splunk btool inputs list --debug
/opt/splunkforwarder/etc/system/default/inputs.conf          [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/system/default/inputs.conf          _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf            host = n00b-splkufw-01.n00blab.local
/opt/splunkforwarder/etc/system/default/inputs.conf          index = _internal

As you can see, the btool output shows me that an input stanza that covers the entire splunk log directory exists in $SPLUNK_HOME/etc/system/default.

You can also confirm the output of:

./splunk list inputstatus

And ensure you see the audit log being monitored

/opt/splunkforwarder/var/log/splunk/audit.log
        file position = 2129
        file size = 2129
        parent = $SPLUNK_HOME/var/log/splunk
        percent = 100.00
        type = open file

The one other thing to check, especially if you are on an older version of UF, is that Splunk UF has internal indexes whitelisted in outputs.conf as part of the SplunkUniversalForwarder App...Here is my output on 6.5.1

./splunk btool outputs list --debug

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .* /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false

It is possible that your UF doesn't have _audit whitelisted...

As long as you have it listed in the ouputs.conf whitelist, and assuming your outputs.conf is configured correctly, all you should need to do is to point your UF and the IDXs and you should start seeing your _internal logs, including audit!

Are you seeing any logs in _internal from this host?

0 Karma

Path Finder

Output from ./splunk list inputstatus. Not sure how how the percent is 100.11. Seem like that might be an indication of a problem...

[splunk@servername bin]$ ./splunk list inputstatus
Cooked:tcp :
        tcp

ExecProcessor:exec commands :
        ./bin/pwchange.sh
                exit status description = exited with code 0
                time closed = 2017-02-15T16:19:43-0600
                time opened = 2017-02-15T16:19:43-0600
                total bytes = 59

Raw:tcp :
        tcp

TailingProcessor:FileStatus :
        $SPLUNK_HOME/etc/splunk.version
                file position = 70
                file size = 70
                percent = 100.00
                type = finished reading

        $SPLUNK_HOME/var/log/splunk
                type = directory

        $SPLUNK_HOME/var/log/splunk/audit.log
                type = directory

        $SPLUNK_HOME/var/log/splunk/metrics.log
                type = directory

        $SPLUNK_HOME/var/log/splunk/splunkd.log
                type = directory

        $SPLUNK_HOME/var/spool/splunk/...stash_new
                type = directory

        $SPLUNK_HOME\var\log\splunk\audit.log
                type = missing

        /opt/splunkforwarder/var/log/splunk/audit.log
                file position = 421540
                file size = 421076
                parent = $SPLUNK_HOME/var/log/splunk/audit.log
                **percent = 100.11**
                type = open file

        /opt/splunkforwarder/var/log/splunk/btool.log
                file position = 392432
                file size = 392432
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/conf.log
                file position = 15181
                file size = 15181
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/first_install.log
                file position = 70
                file size = 70
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/license_usage.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/metrics.log
                file position = 14180368
                file size = 14180368
                parent = $SPLUNK_HOME/var/log/splunk/metrics.log
                percent = 100.00
                type = open file

        /opt/splunkforwarder/var/log/splunk/metrics.log.1
                file position = 25000033
                file size = 25000033
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/metrics.log.2
                file position = 25000065
                file size = 25000065
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/metrics.log.3
                file position = 25000032
                file size = 25000032
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/metrics.log.4
                file position = 25000107
                file size = 25000107
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/metrics.log.5
                file position = 25000122
                file size = 25000122
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/mongod.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/remote_searches.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/scheduler.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/searchhistory.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/splunkd-utility.log
                file position = 40274
                file size = 40274
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/splunkd.log
                file position = 21189839
                file size = 21189839
                parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
                percent = 100.00
                type = open file

        /opt/splunkforwarder/var/log/splunk/splunkd.log.1
                file position = 25000179
                file size = 25000179
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/splunkd_access.log
                file position = 634931
                file size = 634931
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = open file

        /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
                file position = 5939
                file size = 5939
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
                file position = 14343
                file size = 14343
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100.00
                type = finished reading

        /opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
                file position = 0
                file size = 0
                parent = $SPLUNK_HOME/var/log/splunk
                percent = 100
                type = finished reading
0 Karma

Path Finder

I'm running 6.4.5 on server and 6.2.1-6.4.3 on my ufs. I am seeing logs from
/opt/splunkforwarder/var/log/splunk/metrics.log and
/opt/splunkforwarder/var/log/splunk/splunkd.log. On the Windows UFs, the
/opt/splunkforwarder/var/log/splunk/audit.log messages started coming through after I modified inputs.conf. Regardless, I have several outputs.conf files on my linux ufs.

/opt/splunkforwarder/etc/apps/appbaseoutputs/local
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default
/opt/splunkforwarder/etc/system/default
/opt/splunkforwarder/etc/system/local

I don't see anything that seems to indicate that _audit isn't whitelisted.

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf contains:
# Version 6.4.3
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = .*
forwardedindex.2.whitelist = (
audit|introspection|internal)
forwardedindex.filter.disable = false

I will have to have someone on the server team run the splunk commands you mentioned in the morning on the UFs since I don't have access. Hopefully that will shed some light on what is going on. Thanks for the advice.

0 Karma

Path Finder

Below is the output from ./splunk btool outputs list --bebug. I don't see anything that appears to be blacklisting audit.log, but maybe I'm missing something...

[splunk@ufserver bin]$ ./splunk btool outputs list --debug
/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf                          defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf                          [tcpout-server://indexer.domain:9997]
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf           [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf           autoLBFrequency = 40
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf           disabled = false
/opt/splunkforwarder/etc/system/local/outputs.conf                          server = indexer.domain:9997

The output from ./splunk btool inputs list --debug is really to long to put in a comment, but the portion below I think is what is important...

/opt/splunkforwarder/etc/system/default/inputs.conf                        [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = servername.domain
/opt/splunkforwarder/etc/system/default/inputs.conf                        index = _internal
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           [monitor:///opt/splunkforwarder/var/log/splunk/audit.log]
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = servername.domain
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           index = _internal
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = servername.domain
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = servername.domain
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           [monitor:///opt/splunkforwarder\var\log\splunk\audit.log]
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = servername.domain
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf           index = _internal
0 Karma