Splunk Search

Trying to extract from similar events in log file using regex

jd0323fhl
Explorer

I have two distinct events in an application log file: (see below). The events are multiline and seperated by a line of ------------. The fields are not in the same order between the two events. I would like to harmonize the events so that I can report on fields like Timestamp, Message, Category, etc.

I tried to use the automatic field extractor, but it could not read any field other than timestamp. I also attempted to use the following regex patterns, but the fields do not line up. I there a better single regex or a better method using the props.conf/transforms.conf to normalize this data.

Regex 1:
^\w*: (?<timestamp>.*)\s^\w*: (?<message>.*)\s\w*: (?<category>.*)\s\w*: (?<machine>.*)\s\w*\s\w*: (?<app_domain>.*)\s\w*\s\w*: (?<process_name>.*)\s\w*\s\w*:(?<thread_name>[^-]*$)

Event 1:

  Timestamp: 5/29/2018 8:10:46 AM
    Message: Debugging Logging Path: In HttpClientWrapper going to call HttpClientDI.GetAsync(requestUri) with ***"website URL here"***
    Category: Informational
    Machine: 907998-WEB2
    App Domain: /LM/W3SVC/6/ROOT-1-131720684325430305
    Process Name: c:\windows\system32\inetsrv\w3wp.exe
    Thread Name: 

    ----------------------------------------

Regex 2:

^\w*:(?<timestamp>[^\n\r]*)\s\w*:(?<message>.*)\n\w*:(?<category>.*)\n\w*:(?<priority>.*)\n\w*:(?<eventid>.*)\n\w*:(?<severity>.*)\n\w*:(?<title>[^\n\r]*)\n\w*:(?<machine>.*)\n\w*\W\w*:(?<app_domain>[^\n\r]*)\s\w*:(?<process_id>[^\n\r]*)\n\w*\W\w*:(?<process_name>[^\n\r]*)\n\w*\W\w*:(?<thread_name>[^\n\r]*)\n\w*\W\w*:(?<win32_threadid>[^\n\r]*)\n\w*\W\w*:(?<Extended_Properties>[^\n\r]*)$

Event 2:

Timestamp: 5/29/2018 7:46:21 AM
Message: Debugging Logging Path: In HttpClientWrapper going to create authentication token
Category: Informational
Priority: -1
EventId: 0
Severity: Information
Title:
Machine: 907998-WEB2
App Domain: /LM/W3SVC/13/ROOT-1-131720715526291397
ProcessId: 5016
Process Name: c:\windows\system32\inetsrv\w3wp.exe
Thread Name: 
Win32 ThreadId:6056
Extended Properties: LogSource - LeadsWebsite
----------------------------------------
0 Karma

FrankVl
Ultra Champion

I’d suggest to use props and transforms to set up individual extractions for each field. That way the order of the fields does not matter.

Might even work like this (not tested), which takes the part befor the first : as the key and the part after (until end of line) as the value and keeps matching that for each line of the event.

Props.conf

[yoursourcetype]
REPORT-extract-my-fields = extractmyfields

Transforms.conf

[extractmyfields]
REGEX = (?m)^([^:]+):\s+(.*)$
FORMAT = $1::$2
0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...