Splunk Search

Trying to extract from similar events in log file using regex

jd0323fhl
Explorer

I have two distinct events in an application log file: (see below). The events are multiline and seperated by a line of ------------. The fields are not in the same order between the two events. I would like to harmonize the events so that I can report on fields like Timestamp, Message, Category, etc.

I tried to use the automatic field extractor, but it could not read any field other than timestamp. I also attempted to use the following regex patterns, but the fields do not line up. I there a better single regex or a better method using the props.conf/transforms.conf to normalize this data.

Regex 1:
^\w*: (?<timestamp>.*)\s^\w*: (?<message>.*)\s\w*: (?<category>.*)\s\w*: (?<machine>.*)\s\w*\s\w*: (?<app_domain>.*)\s\w*\s\w*: (?<process_name>.*)\s\w*\s\w*:(?<thread_name>[^-]*$)

Event 1:

  Timestamp: 5/29/2018 8:10:46 AM
    Message: Debugging Logging Path: In HttpClientWrapper going to call HttpClientDI.GetAsync(requestUri) with ***"website URL here"***
    Category: Informational
    Machine: 907998-WEB2
    App Domain: /LM/W3SVC/6/ROOT-1-131720684325430305
    Process Name: c:\windows\system32\inetsrv\w3wp.exe
    Thread Name: 

    ----------------------------------------

Regex 2:

^\w*:(?<timestamp>[^\n\r]*)\s\w*:(?<message>.*)\n\w*:(?<category>.*)\n\w*:(?<priority>.*)\n\w*:(?<eventid>.*)\n\w*:(?<severity>.*)\n\w*:(?<title>[^\n\r]*)\n\w*:(?<machine>.*)\n\w*\W\w*:(?<app_domain>[^\n\r]*)\s\w*:(?<process_id>[^\n\r]*)\n\w*\W\w*:(?<process_name>[^\n\r]*)\n\w*\W\w*:(?<thread_name>[^\n\r]*)\n\w*\W\w*:(?<win32_threadid>[^\n\r]*)\n\w*\W\w*:(?<Extended_Properties>[^\n\r]*)$

Event 2:

Timestamp: 5/29/2018 7:46:21 AM
Message: Debugging Logging Path: In HttpClientWrapper going to create authentication token
Category: Informational
Priority: -1
EventId: 0
Severity: Information
Title:
Machine: 907998-WEB2
App Domain: /LM/W3SVC/13/ROOT-1-131720715526291397
ProcessId: 5016
Process Name: c:\windows\system32\inetsrv\w3wp.exe
Thread Name: 
Win32 ThreadId:6056
Extended Properties: LogSource - LeadsWebsite
----------------------------------------
0 Karma

FrankVl
Ultra Champion

I’d suggest to use props and transforms to set up individual extractions for each field. That way the order of the fields does not matter.

Might even work like this (not tested), which takes the part befor the first : as the key and the part after (until end of line) as the value and keeps matching that for each line of the event.

Props.conf

[yoursourcetype]
REPORT-extract-my-fields = extractmyfields

Transforms.conf

[extractmyfields]
REGEX = (?m)^([^:]+):\s+(.*)$
FORMAT = $1::$2
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...