Forward Filtered Splunk Audit Events to Third-Party System

[yujietay]

I would like to forward Splunk audit events containing the keyword "login attempt" to a third-party system.
This is my configuration so far, but it is not working:

props.conf

[audittrail]
TRANSFORMS-routing = forward_audit

transforms.conf

[forward_audit]
REGEX=(login attempt)
DEST_KEY=_SYSLOG_ROUTING
FORMAT=syslog_server

outputs.conf

[syslog:syslog_server]
disabled=false
server=x.x.x.x:514
type=tcp

May I know why?

[micahkemp]

OK, I'm not happy with this answer, but after much testing it's all I have.

The _audit data comes isn't from a "normal" input, it seems. Try to find an inputs.conf stanza that points data to _audit, or a transform that shuffles data into _audit. As best I can tell, they don't exist. I imagine this is because the audittrail data comes from within splunkd and is never logged to a file, and doesn't use anything like a scripted or modular input to push it to splunk. Hence it bypasses props/transforms entirely.

The only workaround I could come up with is to take the opposite approach: forward everything by default via outputs.conf, and use a [default] stanza in props.conf to skip forwarding for everything that matches (which won't happen for audittrail). This method prevents you from further filtering on "login attempt", but I can't find a way to selectively match audittrail events (as per the above paragraph's conclusion).

Here is the config that will forward only audittrail, leaving everything else to only index locally:

outputs.conf

[indexAndForward]
index = true

[syslog]
defaultGroup = syslog_server

[syslog:syslog_server]
server = x.x.x.x:514

props.conf

[default]
# _audit seems to not match [default] so we do this in reverse and send everything that matches to "nowhere"
TRANSFORMS-routing = dont_forward_anything

transforms.conf

[dont_forward_anything]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = nothing

[yujietay]

Hi micahkemp,

The audit logs are in $SPLUNK_HOME/var/log/splunk/audit.log.

[micahkemp]

I do see where I mis-stated the files don't exist, which was due to looking for audittrail in the conf files, instead of audit.log. But, the point remains that the .log file is not used to index the events, and therefore running props/transforms on them will require some form of tinkering.

Thinking more about this now, I'm wondering if my previous comment about blacklisting audit.log from the default monitor stanza, and creating a separate one would be the way to go here. I'm also wondering what else that might break.

[micahkemp]

Yes, but that's not how the events are indexed into Splunk. In system/default/props.conf:

[source::.../var/log/splunk/audit.log(.\d+)?]
TRANSFORMS = send_to_nullqueue
sourcetype = splunk_audit

Also notice that the audit.log format has a prefix on each line that the _audit events in Splunk don't have:

Audit:[timestamp=02-12-2018 15:03:19.917, user=admin, action=search, info=granted REST: /search/jobs/1518447473.9041/control][n/a]

vs

02-12-2018 15:03:35.128 +0000 INFO  AuditLogger - Audit:[timestamp=02-12-2018 15:03:35.128, user=admin, action=search, info=granted REST: /search/jobs/1518447473.9041/control][n/a]

Perhaps it would instead be worth blacklisting audit.log from the catchall [monitor://$SPLUNK_HOME/var/log/splunk] input, and configuring your own input to monitor it instead.

For what it's worth, I sent a link to this answers post to others in the Splunk community, and the only response I got was, "audittrail is weird".

[micahkemp]

How are the events coming in that have the sourcetype audittrail?

As an aside, you don't need the parentheses around login attempt, as you don't use this value later for anything.

[yujietay]

If you search "index=_audit", the audit events that I require have the sourcetype "audittrail".

[micahkemp]

In outputs.conf, you also need to define a default output group, which needs to be a dummy one (which will be overridden by your transform when you actually want to forward the event):

[syslog]
defaultGroup=noForwarding

And if this same Splunk instance also needs to forward, this also needs to go in outputs.conf:

 [indexAndForward]
 index = true

Note: The Splunk docs state defaultGroup isn't required if you don't want to forward by default, but my testing for a recent answers post suggested if it's not set it won't forward at all.

[yujietay]

Hi micahkemp,

Sorry, I only posted a subset of my outputs.conf. I do have the information that you have stated above.

[syslog]
defaultGroup = syslog-output

[indexAndForward]
index=true

[syslog:syslog_server]
disabled=false
server=x.x.x.x:514
type=tcp