Splunk Search
Highlighted

Field Extraction in Message field of Windows Event Log

Path Finder

Hello,

I have an application sending logs to the windows event log with a lognamename of ErrorLogs. The error log looks like this:

05/30/2012 03:17:42 PM

LogName=ErrorLogs

SourceName=LogDataManager

EventCode=0

EventType=4

Type=Information

ComputerName=DEVSERVICE

TaskCategory=%1

OpCode=Info

RecordNumber=31

Keywords=Classic

Message=ApplicationName: EventLogTestWriter

Severity: Informational

Error: Added newline before ApplicationName

StackTrace: N/A

All fields extract well in the autoextraction in search except for the first field 'ApplicationName:' Any ideas? I tried a new line break and while it does send ApplicationName to a new line I still don't get the field to extract.

Highlighted

Re: Field Extraction in Message field of Windows Event Log

Ultra Champion

How does it not 'extract well'?
Is there a field called ApplicationName?
Does it contain anything?
Too much? Too little?

What is the value of the Message field?

/k

0 Karma
Highlighted

Re: Field Extraction in Message field of Windows Event Log

Path Finder

Message itself contains all of ApplicationName through StackTrace inside that field in the eventlog. A splunk search on this yields that the Severity, Error, and StackTrace Fields show up in the 22 interesting fields, I assume being autoextracted, but ApplicationName itself does not get extracted.

0 Karma
Highlighted

Re: Field Extraction in Message field of Windows Event Log

Ultra Champion

So it's pretty much like all the other WinEventLogs, i.e. Message contains everything from the Message= to the end of the event?

The answer is going to be to long for a comment. See answer below.

/k

0 Karma
Highlighted

Re: Field Extraction in Message field of Windows Event Log

Ultra Champion

All WinEvent logs go through the parsing rules defined in $SPLUNK_HOME/etc/system/local/props.conf and $SPLUNK_HOME/etc/system/local/transforms.conf.

These state the following;

props.conf:

[source::WinEventLog...]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none

etc, etc

transforms.conf:

[wel-message]
REGEX = (?sm)^(?<_pre_msg>.+)\nMessage=(?<Message>.+)$
CLEAN_KEYS = false

[wel-eq-kv]
SOURCE_KEY = _pre_msg
DELIMS     = "\n","="
MV_ADD     = true

[wel-col-kv]
SOURCE_KEY = Message
REGEX      = \n([^:\n\r]+):[ \t]++([^\n]*)
FORMAT     = $1::$2
MV_ADD     = true

This means that anything with the source of WinEventLog:somthing will be run through the three field extracting transforms wel-message, wel-eq-kv and wel-col-kv. In that order.

wel-message splits the event into two fields, _pre_msg and Message.
wel-eq-kv splits the _pre_msg into field/value pairs based on 'equals' (=).
wel-col-kv splits the Message into field/value pairs based on colons (:).

However the REGEX in wel-col-kv requires that a newline preceeds the first capture group, and that newline does not exist in your first row.

Perhaps you could/should create new props.conf/transforms.conf stanzas in the /$SPLUNK_HOME/etc/system/local directory, which would be very similar, but the differently named. The props-stanza (from which the transforms are called) should be more specific, i.e. [WinEventLog:ErrorLogs], and the transforms that are called through the REPORT should be called welel-message, welel-eq-kv and welel-col-kv respectively (. Then the REGEX in welel-col-kv should have the leading newline made optional (by a question mark);

 [welel-col-kv]
 SOURCE_KEY = Message
 REGEX      = \n?([^:\n\r]+):[ \t]++([^\n]*)
 FORMAT     = $1::$2
 MV_ADD     = true

Hope this helps, at least a little bit.

Kristian

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.