Dynatrace audit logs indexing problem

[splunkreal]

Hello,

we try to index correctly SecAudit-BackendServer.1.log from Dynatrace however the non-encrypted log files have special characters just before the timestamp :

\x00\x00\x00\xEB\x00\x00\x00
2018-08-14T16:34:51.920+0200 user=toto,source=1.2.3.4,category=AuditLog,object=,event=Access,status=success,message="successfully read audit log /opt/dynatrace/dynatrace-7.0/log/server/SecAudit-FrontendServer.1.log"

in ssh:

How would you handle with TIME_PREFIX in props.conf?

Thanks.

[sudosplunk]

Hi @realsplunk,

IMO, this should be handled by using LINE_BREAKER. Configure line breaking to discard all special characters before date, something like below

props.conf

[sourcetype]
LINE_BREAKER = ([\r\n]+^.+)\d{4}\-\d{2}\-\d{1,2} ## This will discard newline, carriage return characters along with encrypted text.
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

[splunkreal]

Hi Nittala_surya, I'm in contact with support because this doesn't work maybe bad characters impact so I'm testing SEDCMD to clean data first.

[splunkreal]

Thanks, however as you can see there are special characters, how would you write regex with :

beginning with anything (risky?) until YYYY-MM-DD?

LINE_BREAKER = (^.+)\d{4}-\d{2}-\d{1,2} ?

https://regex101.com/r/6fN7JB/1

Thanks 🙂

[sudosplunk]

Your LINE_BREAKING regex should also include \n-new line and \r-carriage return characters inside capturing group.

Per docs,

The LINE_BREAKER expression must contain a capturing group (a pair of parentheses that defines an identified subcomponent of the match.)

Wherever the expression matches, Splunk software considers the start of the first capturing group to be the end of the previous event, and considers the end of the first capturing group to be the start of the next event.

Splunk software discards the contents of the first capturing group. This content will not be present in any event, as Splunk software considers this text to come between lines.

That said, LINE_BREAKER = ([\r\n]+^.+)\d{4}\-\d{2}\-\d{1,2} will discard any special characters until YYYY-MM-DD.

https://regex101.com/r/6fN7JB/2