Splunk Universal Forwarders audit logs merged - audit.conf configuration

[edoardo_vicendo]

Hi All,

As indicated here (https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-monitor-SPLUNK-HOME-var-log-splun...), I have been able to get the audit.log from our Universal Forwarders with audittrail sourcetype.

Unfortunately sometimes those events read from $SPLUNK_HOME/var/log/splunk/audit.log are merged in a unique event (even if each event is in a new line and starts with a timestamp).

In our deployment we have Universal Forwarders sending data to Heavy Forwarders that then send them to Indexers:

UF --> HF --> IDX

What I tried to do is to deploy a props.conf on the HF to indicate the following:

[audittrail]
SHOULD_LINEMERGE = false
SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO  AuditLogger - //g

But even the SEDCMD is not applied.

And I can see with the following command that the configuration are properly read in the HF:

splunk btool props list --debug

Due to that I tried adding this props.conf directly on the UF and it is working (but it is not a good solution for us because we don't want to force the local processing on the UF).

[audittrail]
SHOULD_LINEMERGE = false
SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO  AuditLogger - //g
force_local_processing = true

I believe the issue is related to the fact that the audit logs from UF are sent to HF indexQueue instead of parsingQueue. 

I tried also to add the audit.conf file both in UF and HF as follow without any luck:

[default]
queueing=false

Reading further on Splunk documentation (https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Auditconf) :

queueing = <boolean>
* Whether or not audit events are sent to the indexQueue.
* If set to "true", audit events are sent to the indexQueue.
* If set to "false", you must add an inputs.conf stanza to tail the
  audit log for the events reach your index.
* Default: true

My questions are:

• Do you know what does it means "If set to "false", you must add an inputs.conf stanza to tail the audit log for the events reach your index."
• Do you have any idea on how to apply on the HF the props.conf to the audit events coming from the UF without having to deploy it directly on UF with  force_local_processing=true

Thanks a lot,
Edoardo

[edoardo_vicendo · Accepted solution]

I have been able to solve this with Splunk support.

So basically there are 2 options:

• OPTION 1 - enable UF audit.log monitoring + force local processing on UF

This option increase a little bit the cpu consumption on UF because you have to parse the events directly on UF, but it will be enabled only for this specific sourcetype and usually few audit events per day on each UF are generated.

On UF:

myapp/local/inputs.conf

# Specific configuration to enable monitoring Splunk Universal Forwarder audit logs
# by default they are sent to null queue

#*nix
[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = audittrail
source = audittrail

#Windows
[monitor://$SPLUNK_HOME\var\log\splunk\audit.log]
index = _audit
sourcetype = audittrail
source = audittrail

myapp/local/props.conf

[audittrail]
SHOULD_LINEMERGE = false
force_local_processing = true

• OPTION 2 - enable UF audit.log monitoring + enable HF event parsing

This has been tested only on *nix

On UF (local/inputs.conf)

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = ufw_audittrail
source = ufw_audittrail

on HF (local/props.conf)

[ufw_audittrail]
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%l %z
TIME_PREFIX = ^

Best Regards,
Edoardo