I created an app named a_uf_inputs_conf. The app simply contains inputs.conf that has the monitor stanza's below. This app was deployed to both Windows and Linux servers. It is working on the Windows servers, but not on the Linux servers.
[monitor://$SPLUNK_HOME\var\log\splunk\audit.log]
_TCP_ROUTING = *
index = _internal
[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
_TCP_ROUTING = *
index = _internal
I see log messages from splunkd.log that seem to indicate that Splunk is attempting to monitor the files, but when I search the _internal index for host=servername.domain and source="/opt/splunkforwarder/var/log/splunk/splunkd.log" i get no results.
02-14-2017 08:31:07.903 -0600 INFO WatchedFile - Will begin reading at offset=111468 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 21:29:25.417 -0600 INFO WatchedFile - Will begin reading at offset=108519 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 16:37:32.705 -0600 INFO WatchedFile - Will begin reading at offset=105392 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 14:40:29.015 -0600 INFO WatchedFile - Will begin reading at offset=103010 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 14:02:01.743 -0600 INFO WatchedFile - Will begin reading at offset=100628 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
02-13-2017 13:28:58.577 -0600 INFO WatchedFile - Will begin reading at offset=96238 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.host = servername.domain source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
I have verified that the app (inputs.conf file) is being deployed. The file permissions are okay. I'm not really sure what I should check next?
By default, Splunk audit events coming from non-Windows universal forwarders are sent to the null queue by way of system default configuration in props.conf.
#system/default/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit
As a workaround, we deployed an app with a local props.conf and referenced a non-existing transform to override the default of send_to _nullqueue .
#spp_cluster_indexer_base
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza
sourcetype = splunk_audit
By default, Splunk audit events coming from non-Windows universal forwarders are sent to the null queue by way of system default configuration in props.conf.
#system/default/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit
As a workaround, we deployed an app with a local props.conf and referenced a non-existing transform to override the default of send_to _nullqueue .
#spp_cluster_indexer_base
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza
sourcetype = splunk_audit
Hi,
I tried with the posted solution but unfortunately it was not successful, therefore I worked directly with the Splunk support team to solve this, here below our findings.
Basically the audit.log from the Splunk Universal Forwarders is not indexed due to the fact that there is a default configuration that send it to null queue:
default/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit
even overwriting it with a local stanza was not working.
local/props.conf
[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = nostanza
sourcetype = splunk_audit
In order to solve it, it is needed to bypass the default configuration that is sending to null queue with a monitor stanza in inputs.conf, therefore I created a specific app in the Deployment Server and deployed to all the Splunk Universal Forwarders (*nix and Windows):
myapp/local/inputs.conf
# Specific configuration to enable monitoring Splunk Universal Forwarder audit logs
# by default they are sent to null queue
#*nix
[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = audittrail
source = audittrail
#Windows
[monitor://$SPLUNK_HOME\var\log\splunk\audit.log]
index = _audit
sourcetype = audittrail
source = audittrail
In this way the Splunk Universal Forwarders audit logs will be indexed in the same index and with the same source and sourcetype of the one coming from Splunk Enterprise servers (Search Heads, Indexers, Master Node, Deployment Server, Heavy Forwarders, License Master etc...).
NOTE:
sourcetype --> it need to be audittrail otherwise data are not ingested
source --> it need to be different from $SPLUNK_HOME/var/log/splunk/audit.log otherwise data are not ingested
I hope this can help you!
Best Regards,
Edoardo
Here my answer on another thread to solve, once audit.log was ingested, the issue with merged events:
Ah! After re-reading your question, the only thing I forgot to ask in my answer above was:
did you check index=_audit
?
On my nix system it throws my local audit logs in _audit, while my MS events are still in _internal
http://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Searchforauditevents
I did a search for the last 90 ( in our Production environment ) of index=_audit. All the results I find are from Linux servers running the Splunk server application. I don't find anything from a Linux server running the Splunk Universal Forwarder application.
I do think you are correct about audit.log being monitored by default, because I searched the Last 90 days (in a different environment) index=_internal sourcetype=splunk_audit
and found 100's of Windows hosts logging messages from C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log.
The Production environment that I searched has 76 linux servers running the Universal Forwarder application. I found an /opt/splunkforwarder/var/log/splunk/audit.log/audit.log file with entries during that time frame.
yeah thats for sure the default and thats what I see for my windows host.
Looks like in my enviro the nix logs get flipped over to index=_audit sourcetype=audittrail
which is a sourcetype that is in the CIM app. Do you see anything there? You are an admin user right?
I am in our Splunk admins AD group, but just to be certain I logged in as the user admin and searched index=_audit sourcetype=audittrail for the last 90 days.
The search returned 16,656,224 events, but the hosts were all splunk servers. Not anything from a host with the Universal Forwarder installed.
hmmm.....and I assume you have checked for index=* host=theforwarderyouarelookingfor source=*audit.log
?
Also have you checked locally on the UF to ensure there actually is any audit activity?
When I search index=* host=theforwarderiamlookingfor , I only get results from index=os , which is the index that our /var/log/secure goes to. Does searching index=* search _internal _audit, etc?
I found a Linux forwarder with log entries for audit.log in the last 90 days. I have a case open with support. If we get to the bottom of this, I will update this question with the resolution.
Hi matthewroberson69!
You actually shouldn't need to create a monitor for the audit log, as Splunk monitors it out of the box!
you can use btool to examine the inputs on your UF:
[splunker@n00b-splkufw-01 bin]$ ./splunk btool inputs list --debug
/opt/splunkforwarder/etc/system/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = n00b-splkufw-01.n00blab.local
/opt/splunkforwarder/etc/system/default/inputs.conf index = _internal
As you can see, the btool output shows me that an input stanza that covers the entire splunk log directory exists in $SPLUNK_HOME/etc/system/default.
You can also confirm the output of:
./splunk list inputstatus
And ensure you see the audit log being monitored
/opt/splunkforwarder/var/log/splunk/audit.log
file position = 2129
file size = 2129
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file
The one other thing to check, especially if you are on an older version of UF, is that Splunk UF has internal indexes whitelisted in outputs.conf as part of the SplunkUniversalForwarder App...Here is my output on 6.5.1
./splunk btool outputs list --debug
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .* /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
It is possible that your UF doesn't have _audit whitelisted...
As long as you have it listed in the ouputs.conf whitelist, and assuming your outputs.conf is configured correctly, all you should need to do is to point your UF and the IDXs and you should start seeing your _internal logs, including audit!
Are you seeing any logs in _internal from this host?
Output from ./splunk list inputstatus. Not sure how how the percent is 100.11. Seem like that might be an indication of a problem...
[splunk@servername bin]$ ./splunk list inputstatus
Cooked:tcp :
tcp
ExecProcessor:exec commands :
./bin/pwchange.sh
exit status description = exited with code 0
time closed = 2017-02-15T16:19:43-0600
time opened = 2017-02-15T16:19:43-0600
total bytes = 59
Raw:tcp :
tcp
TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading
$SPLUNK_HOME/var/log/splunk
type = directory
$SPLUNK_HOME/var/log/splunk/audit.log
type = directory
$SPLUNK_HOME/var/log/splunk/metrics.log
type = directory
$SPLUNK_HOME/var/log/splunk/splunkd.log
type = directory
$SPLUNK_HOME/var/spool/splunk/...stash_new
type = directory
$SPLUNK_HOME\var\log\splunk\audit.log
type = missing
/opt/splunkforwarder/var/log/splunk/audit.log
file position = 421540
file size = 421076
parent = $SPLUNK_HOME/var/log/splunk/audit.log
**percent = 100.11**
type = open file
/opt/splunkforwarder/var/log/splunk/btool.log
file position = 392432
file size = 392432
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/conf.log
file position = 15181
file size = 15181
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/first_install.log
file position = 70
file size = 70
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/license_usage.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
/opt/splunkforwarder/var/log/splunk/metrics.log
file position = 14180368
file size = 14180368
parent = $SPLUNK_HOME/var/log/splunk/metrics.log
percent = 100.00
type = open file
/opt/splunkforwarder/var/log/splunk/metrics.log.1
file position = 25000033
file size = 25000033
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/metrics.log.2
file position = 25000065
file size = 25000065
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/metrics.log.3
file position = 25000032
file size = 25000032
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/metrics.log.4
file position = 25000107
file size = 25000107
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/metrics.log.5
file position = 25000122
file size = 25000122
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/mongod.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
/opt/splunkforwarder/var/log/splunk/remote_searches.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
/opt/splunkforwarder/var/log/splunk/scheduler.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
/opt/splunkforwarder/var/log/splunk/searchhistory.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
file position = 40274
file size = 40274
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/splunkd.log
file position = 21189839
file size = 21189839
parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
percent = 100.00
type = open file
/opt/splunkforwarder/var/log/splunk/splunkd.log.1
file position = 25000179
file size = 25000179
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/splunkd_access.log
file position = 634931
file size = 634931
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file
/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
file position = 5939
file size = 5939
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
file position = 14343
file size = 14343
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading
/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading
I'm running 6.4.5 on server and 6.2.1-6.4.3 on my ufs. I am seeing logs from
/opt/splunkforwarder/var/log/splunk/metrics.log and
/opt/splunkforwarder/var/log/splunk/splunkd.log. On the Windows UFs, the
/opt/splunkforwarder/var/log/splunk/audit.log messages started coming through after I modified inputs.conf. Regardless, I have several outputs.conf files on my linux ufs.
/opt/splunkforwarder/etc/apps/app_base_outputs/local
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default
/opt/splunkforwarder/etc/system/default
/opt/splunkforwarder/etc/system/local
I don't see anything that seems to indicate that _audit isn't whitelisted.
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf contains:
# Version 6.4.3
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false
I will have to have someone on the server team run the splunk commands you mentioned in the morning on the UFs since I don't have access. Hopefully that will shed some light on what is going on. Thanks for the advice.
Below is the output from ./splunk btool outputs list --bebug. I don't see anything that appears to be blacklisting audit.log, but maybe I'm missing something...
[splunk@ufserver bin]$ ./splunk btool outputs list --debug
/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://indexer.domain:9997]
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf autoLBFrequency = 40
/opt/splunkforwarder/etc/apps/app_base_outputs/local/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/local/outputs.conf server = indexer.domain:9997
The output from ./splunk btool inputs list --debug is really to long to put in a comment, but the portion below I think is what is important...
/opt/splunkforwarder/etc/system/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = servername.domain
/opt/splunkforwarder/etc/system/default/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/audit.log]
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = servername.domain
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = servername.domain
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = servername.domain
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf index = _internal
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf [monitor:///opt/splunkforwarder\var\log\splunk\audit.log]
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf _TCP_ROUTING = *
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/local/inputs.conf host = servername.domain
/opt/splunkforwarder/etc/apps/a_uf_inputs_conf/local/inputs.conf index = _internal