topic Re: Read timezone from openvas logs? Regex, strftime, won't detect it. in Splunk Search

Read timezone from openvas logs? Regex, strftime, won't detect it.

JSkier — Tue, 24 Apr 2018 21:30:43 GMT

I'm inputting openvas logs into splunk. Works great for .messages, not so much for .log files. Below is how the lines look:

Info<18 chars pos>2018-04-23 23h04.55 utc:31730 SOME MESSAGE
Info<18 chars pos>2018-04-24 10h25.34 CDT:539 SOME MESSAGE

So, time zone starts at the 18 character. Splunk cannot read it no matter how far I go.

Timestamp prefix regex:
^.{18}\K
I also tried the more pythonic way of doing this regex:
^.{18}
Still can't get it detected in splunk, so I added strftime:
%Y-%M-%d %Hh%M.%S %Z:%s

Any ideas on how to get the timestamp recognized correctly in splunk?

Re: Read timezone from openvas logs? Regex, strftime, won't detect it.

woodcock — Tue, 24 Apr 2018 22:42:10 GMT

The correct settings on your indexers would be this (in #props.conf):

TIME_PREFIX = chars\s+pos\>
TIME_FORMAT = %Y-%m-%d %Hh%m.%S %Z:%N
MAX_TIMESTAMP_LOOKAHEAD = 29

Re: Read timezone from openvas logs? Regex, strftime, won't detect it.

woodcock — Tue, 24 Apr 2018 22:43:07 GMT

Your mistake was using %M twice; that is totally invalid.

Re: Read timezone from openvas logs? Regex, strftime, won't detect it.

JSkier — Wed, 25 Apr 2018 13:18:07 GMT

This got me on the right path, thank you!

UPDATE: Removed %N, it is not actually seconds.
UPDATE2: Turns out some timestamps start 19 chars in, I have updated the props below to reflect this. The log source timestamps are a mess (some local tz, others utc), splunk isn't able to differentiate them, so I removed the TZ from the format. It will require a few source types to get it to a very clean place.

Timezone stuff is still weird, so it will need more tuning there on my end, but this ended up working for parsing:

SHOULD_LINEMERGE = false
TIME_PREFIX = ^\w+\s{1,5}\w+:(\s+\w+|\w+):
TIME_FORMAT = %Y-%m-%d %Hh%M.%S