Splunk Universal Forwarders audit logs merged - audit.conf configuration

edoardo_vicendo — Thu, 02 Jul 2020 09:12:11 GMT

Hi All,

As indicated here (https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-monitor-SPLUNK-HOME-var-log-splunk-audit-log/m-p/506185#M86203), I have been able to get the audit.log from our Universal Forwarders with audittrail sourcetype.

Unfortunately sometimes those events read from $SPLUNK_HOME/var/log/splunk/audit.log are merged in a unique event (even if each event is in a new line and starts with a timestamp).

In our deployment we have Universal Forwarders sending data to Heavy Forwarders that then send them to Indexers:

UF --> HF --> IDX

What I tried to do is to deploy a props.conf on the HF to indicate the following:

[audittrail] SHOULD_LINEMERGE = false SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO AuditLogger - //g

But even the SEDCMD is not applied.

And I can see with the following command that the configuration are properly read in the HF:

splunk btool props list --debug

Due to that I tried adding this props.conf directly on the UF and it is working (but it is not a good solution for us because we don't want to force the local processing on the UF).

[audittrail] SHOULD_LINEMERGE = false SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO AuditLogger - //g force_local_processing = true

I believe the issue is related to the fact that the audit logs from UF are sent to HF indexQueue instead of parsingQueue.

I tried also to add the audit.conf file both in UF and HF as follow without any luck:

[default] queueing=false

Reading further on Splunk documentation (https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Auditconf) :

queueing = <boolean> * Whether or not audit events are sent to the indexQueue. * If set to "true", audit events are sent to the indexQueue. * If set to "false", you must add an inputs.conf stanza to tail the audit log for the events reach your index. * Default: true

My questions are:

Do you know what does it means "If set to "false", you must add an inputs.conf stanza to tail the audit log for the events reach your index."
Do you have any idea on how to apply on the HF the props.conf to the audit events coming from the UF without having to deploy it directly on UF with force_local_processing=true

Thanks a lot,
Edoardo

Re: Splunk Universal Forwarders audit logs merged - audit.conf configuration

edoardo_vicendo — Thu, 16 Jul 2020 13:23:24 GMT

I have been able to solve this with Splunk support.

So basically there are 2 options:

OPTION 1 - enable UF audit.log monitoring + force local processing on UF

This option increase a little bit the cpu consumption on UF because you have to parse the events directly on UF, but it will be enabled only for this specific sourcetype and usually few audit events per day on each UF are generated.

On UF:

myapp/local/inputs.conf

# Specific configuration to enable monitoring Splunk Universal Forwarder audit logs # by default they are sent to null queue #*nix [monitor://$SPLUNK_HOME/var/log/splunk/audit.log] index = _audit sourcetype = audittrail source = audittrail #Windows [monitor://$SPLUNK_HOME\var\log\splunk\audit.log] index = _audit sourcetype = audittrail source = audittrail

myapp/local/props.conf

[audittrail] SHOULD_LINEMERGE = false force_local_processing = true

OPTION 2 - enable UF audit.log monitoring + enable HF event parsing

This has been tested only on *nix

On UF (local/inputs.conf)

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log] index = _audit sourcetype = ufw_audittrail source = ufw_audittrail

on HF (local/props.conf)

[ufw_audittrail] LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false TIME_FORMAT = %m-%d-%Y %H:%M:%S.%l %z TIME_PREFIX = ^

Best Regards,
Edoardo

topic Splunk Universal Forwarders audit logs merged - audit.conf configuration in Getting Data In

Splunk Universal Forwarders audit logs merged - audit.conf configuration

Re: Splunk Universal Forwarders audit logs merged - audit.conf configuration