Splunk Search

Trying to extract from similar events in log file using regex


I have two distinct events in an application log file: (see below). The events are multiline and seperated by a line of ------------. The fields are not in the same order between the two events. I would like to harmonize the events so that I can report on fields like Timestamp, Message, Category, etc.

I tried to use the automatic field extractor, but it could not read any field other than timestamp. I also attempted to use the following regex patterns, but the fields do not line up. I there a better single regex or a better method using the props.conf/transforms.conf to normalize this data.

Regex 1:
^\w*: (?<timestamp>.*)\s^\w*: (?<message>.*)\s\w*: (?<category>.*)\s\w*: (?<machine>.*)\s\w*\s\w*: (?<app_domain>.*)\s\w*\s\w*: (?<process_name>.*)\s\w*\s\w*:(?<thread_name>[^-]*$)

Event 1:

  Timestamp: 5/29/2018 8:10:46 AM
    Message: Debugging Logging Path: In HttpClientWrapper going to call HttpClientDI.GetAsync(requestUri) with ***"website URL here"***
    Category: Informational
    Machine: 907998-WEB2
    App Domain: /LM/W3SVC/6/ROOT-1-131720684325430305
    Process Name: c:\windows\system32\inetsrv\w3wp.exe
    Thread Name: 


Regex 2:


Event 2:

Timestamp: 5/29/2018 7:46:21 AM
Message: Debugging Logging Path: In HttpClientWrapper going to create authentication token
Category: Informational
Priority: -1
EventId: 0
Severity: Information
Machine: 907998-WEB2
App Domain: /LM/W3SVC/13/ROOT-1-131720715526291397
ProcessId: 5016
Process Name: c:\windows\system32\inetsrv\w3wp.exe
Thread Name: 
Win32 ThreadId:6056
Extended Properties: LogSource - LeadsWebsite
0 Karma

Ultra Champion

I’d suggest to use props and transforms to set up individual extractions for each field. That way the order of the fields does not matter.

Might even work like this (not tested), which takes the part befor the first : as the key and the part after (until end of line) as the value and keeps matching that for each line of the event.


REPORT-extract-my-fields = extractmyfields


REGEX = (?m)^([^:]+):\s+(.*)$
FORMAT = $1::$2
0 Karma
Get Updates on the Splunk Community!

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...