Trying to extract from similar events in log file using regex

jd0323fhl — Fri, 01 Jun 2018 20:08:55 GMT

I have two distinct events in an application log file: (see below). The events are multiline and seperated by a line of ------------. The fields are not in the same order between the two events. I would like to harmonize the events so that I can report on fields like Timestamp, Message, Category, etc.

I tried to use the automatic field extractor, but it could not read any field other than timestamp. I also attempted to use the following regex patterns, but the fields do not line up. I there a better single regex or a better method using the props.conf/transforms.conf to normalize this data.

Regex 1:
^\w*: (?<timestamp>.*)\s^\w*: (?<message>.*)\s\w*: (?<category>.*)\s\w*: (?<machine>.*)\s\w*\s\w*: (?<app_domain>.*)\s\w*\s\w*: (?<process_name>.*)\s\w*\s\w*:(?<thread_name>[^-]*$)

Event 1:

  Timestamp: 5/29/2018 8:10:46 AM
    Message: Debugging Logging Path: In HttpClientWrapper going to call HttpClientDI.GetAsync(requestUri) with ***"website URL here"***
    Category: Informational
    Machine: 907998-WEB2
    App Domain: /LM/W3SVC/6/ROOT-1-131720684325430305
    Process Name: c:\windows\system32\inetsrv\w3wp.exe
    Thread Name: 

    ----------------------------------------

Regex 2:

^\w*:(?<timestamp>[^\n\r]*)\s\w*:(?<message>.*)\n\w*:(?<category>.*)\n\w*:(?<priority>.*)\n\w*:(?<eventid>.*)\n\w*:(?<severity>.*)\n\w*:(?<title>[^\n\r]*)\n\w*:(?<machine>.*)\n\w*\W\w*:(?<app_domain>[^\n\r]*)\s\w*:(?<process_id>[^\n\r]*)\n\w*\W\w*:(?<process_name>[^\n\r]*)\n\w*\W\w*:(?<thread_name>[^\n\r]*)\n\w*\W\w*:(?<win32_threadid>[^\n\r]*)\n\w*\W\w*:(?<Extended_Properties>[^\n\r]*)$

Event 2:

Timestamp: 5/29/2018 7:46:21 AM
Message: Debugging Logging Path: In HttpClientWrapper going to create authentication token
Category: Informational
Priority: -1
EventId: 0
Severity: Information
Title:
Machine: 907998-WEB2
App Domain: /LM/W3SVC/13/ROOT-1-131720715526291397
ProcessId: 5016
Process Name: c:\windows\system32\inetsrv\w3wp.exe
Thread Name: 
Win32 ThreadId:6056
Extended Properties: LogSource - LeadsWebsite
----------------------------------------

Re: Trying to extract from similar events in log file using regex

FrankVl — Sat, 02 Jun 2018 01:12:31 GMT

I’d suggest to use props and transforms to set up individual extractions for each field. That way the order of the fields does not matter.

Might even work like this (not tested), which takes the part befor the first : as the key and the part after (until end of line) as the value and keeps matching that for each line of the event.

Props.conf

[yoursourcetype]
REPORT-extract-my-fields = extractmyfields

Transforms.conf

[extractmyfields]
REGEX = (?m)^([^:]+):\s+(.*)$
FORMAT = $1::$2

topic Trying to extract from similar events in log file using regex in Splunk Search

Trying to extract from similar events in log file using regex

Re: Trying to extract from similar events in log file using regex