topic Re: How to extract fields from the message of a windows event log? in All Apps and Add-ons

How to extract fields from the message of a windows event log?

john0499 — Tue, 29 Sep 2020 14:07:47 GMT

Hi all,

I have a problem extracting fields after the Message= portion of a windows application event log. Everything after Message= is being added as single value to the message field.

I think it may be the due to the formatting of the data after Message=, which seems to have a leading space or tab. It looks like this:

Message=Process started in passive mode.

Command Line: C:\something\something\something.exe
Process Id: 8988
...

(the the white space is being removed from this post - its in front of 'Command' and 'Process')

There are about 50 more fields after these, all on new lines but with a space or tab before them.

I have tried playing with the regex from transforms.conf below, but I've only been able to move the problem from the message field to a new field called Command_Line that contains a value of everything below it.

[wel-col-kv]
SOURCE_KEY = Message
REGEX = \n([^:\n\r]+):[ \t]++([^\n]*)
FORMAT = $1::$2
MV_ADD = true

I have found some similar questions asked but haven't been able to come up with a solution yet. Any help would be greatly appreciated.

Re: How to extract fields from the message of a windows event log?

adonio — Wed, 24 May 2017 02:48:52 GMT

are you using the Windows TA?
https://splunkbase.splunk.com/app/742/

Re: How to extract fields from the message of a windows event log?

john0499 — Mon, 29 May 2017 23:46:31 GMT

I do have the TA, although I don't believe it's actually needed as windows event extraction is included in the default transforms.conf

I managed to get it working with the following changes to the default:

REGEX = \n? \t:[ \t]++([^\r]*)

FYI this was to extract avecto defendpoint events

Re: How to extract fields from the message of a windows event log?

jwindley_splunk — Tue, 27 Nov 2018 10:01:05 GMT

Hi John, I have the same problem parsing the avecto logs from Message. I tried your new regex but it didn't solve it for me. Strangely though the default regex parses everything fine for me in regex101. Did you make any other changes at all other than to the wel-col-kv settings?

Thanks

Re: How to extract fields from the message of a windows event log?

john0499 — Tue, 29 Sep 2020 22:14:23 GMT

Hi jwindley,

this is what I have in my avecto app:

props.conf
[source::WinEventLog:Application]
REPORT-MESSAGE = avec-wel-message, avec-wel-eq-kv, avec-wel-col-kv
KV_MODE=none

Note the below settings are effectively legacy, in place here to handle

data coming from much much older forwarders (3.x & 4.x)

SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = (\r\n)
TRANSFORMS-FIELDS = strip-winevt-linebreaker

transforms.conf
[avec-wel-message]
REGEX = (?sm)^(?<_pre_msg>.+)\nMessage=(?.+)$
CLEAN_KEYS = false

[avec-wel-eq-kv]
SOURCE_KEY = _pre_msg
DELIMS = "\n","="
MV_ADD = true

[avec-wel-col-kv]
SOURCE_KEY = Message
REGEX = \n? \t:[ \t]++([^\r]*)
FORMAT = $1::$2
MV_ADD = true