Getting Data In

Help in parsing logs- How to cut out or blacklist the first portion and only grabbing the col name?

jcgever
Explorer

Been trying to work out the regex for the props and transforms for onbase logging and I keep hitting a brick wall. Here is a sample log of what I have been working on:

<?xml version="1.0" encoding="utf-8"?>
<diagnosticsLog type="error-profile" startDate="07/28/2022 01:10:20">
<!--Build 18.0.1.42-->
<columns>
<column friendlyName="time" name="time" />
<column friendlyName="Result" name="Result" />
<column friendlyName="Module" name="Module" />
<column friendlyName="Class" name="Class" />
<column friendlyName="SourceFile" name="SourceFile" />
<column friendlyName="Method" name="Method" />
<column friendlyName="SourceLine" name="SourceLine" />
<column friendlyName="Severity" name="Severity" />
<column friendlyName="MachineName" name="MachineName" />
<column friendlyName="IpAddress" name="IpAddress" />
<column friendlyName="ErrorId" name="ErrorId" />
<column friendlyName="ProcessID" name="ProcessID" />
<column friendlyName="ThreadID" name="ThreadID" />
<column friendlyName="TimeSpan" name="TimeSpan" />
<column friendlyName="User" name="User" />
<column friendlyName="HTTPSessionID" name="HTTPSessionID" />
<column friendlyName="HTTPForward" name="HTTPForward" />
<column friendlyName="SessionID" name="SessionID" />
<column friendlyName="SessionGUID" name="SessionGUID" />
<column friendlyName="Datasource" name="Datasource" />
<column friendlyName="Sequence" name="Sequence" />
<column friendlyName="LocalSequence" name="LocalSequence" />
<column friendlyName="Message" name="Message" />
<column friendlyName="AppPoolName" name="AppPoolName" />
</columns>
<rows>
<row>
<col name="time">07/28/2022 01:10:20</col>
<col name="TimeSpan">N/A</col>
<col name="ThreadID">0x0000000A</col>
<col name="User"></col>
<col name="HTTPSessionID"></col>
<col name="HTTPForward"></col>
<col name="SessionGUID"></col>
<col name="SessionID">0</col>
<col name="Datasource">OnBaseQA</col>
<col name="AppPoolName"></col>
<col name="IpAddress"></col>
<col name="MachineName"></col>
<col name="Result">0x00000000</col>
<col name="Message">FileLoadException: File [D:\Program Files\Hyland\Services\Distribution\System.Runtime.CompilerServices.Unsafe.dll] Message [Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=4.0.3.0, Culture=neutral, PublicKeyToken=abcd1234' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x901234)]</col>
<col name="Module">Hyland.Core</col>
<col name="Class">AssemblyRegistration</col>
<col name="Method">ReadAssemblyXml</col>
<col name="SourceFile"></col>
<col name="SourceLine">0</col>
<col name="Severity">Error</col>
<col name="ErrorId"></col>
</row>

I'm trying to cut out or blacklist the first portion with the column naming and only grabbing the col name in between the "" with the corresponding data. I'm very new to manipulating data like this and am just starting to understand regex. These are my attempts on props and transforms:

 

Transforms:

[log_time]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"

[time_span]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"

[thread_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"

[user]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"

[http_session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"

[http_forward]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"

[session_guid]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"

[session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"

[datasource]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"

[app_pool_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"

[ip_address]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"

[machine_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"

[result]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"

[message]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"

[module]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"

[class]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"

[method]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"

[source_file]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"

[source_line]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"

[severity]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"

[error_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"

 

Props:
[error_profile]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
TRANSFORMS-set = log_time, time_span, thread_id, user, http_session_id, http_forward, session_guid, session_id, datasource, app_pool_name, ip_address, machine_name, result, message, module, class, method, source_file, source_line, severity, error_id

Labels (2)
Tags (2)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @jcgever,

I didn't check all your regex syntax but seems ok. Your transforms are wrong should be like below. 

[log_time]
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"
WRITE_META = true

[time_span]
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"
WRITE_META = true

[thread_id]
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"
WRITE_META = true

[user]
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"
WRITE_META = true

[http_session_id]
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"
WRITE_META = true

[http_forward]
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"
WRITE_META = true

[session_guid]
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"
WRITE_META = true

[session_id]
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"
WRITE_META = true

[datasource]
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"
WRITE_META = true

[app_pool_name]
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"
WRITE_META = true

[ip_address]
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"
WRITE_META = true

[machine_name]
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"
WRITE_META = true

[result]
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"
WRITE_META = true

[message]
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"
WRITE_META = true

[module]
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"
WRITE_META = true

[class]
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"
WRITE_META = true

[method]
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"
WRITE_META = true

[source_file]
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"
WRITE_META = true

[source_line]
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"
WRITE_META = true

[severity]
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"
WRITE_META = true

[error_id]
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"
WRITE_META = true

Also you must add these fields to fields.conf in your search head in order to be able to search;

[log_time]
INDEXED_VALUE = false

[time_span]
INDEXED_VALUE = false

[thread_id]
INDEXED_VALUE = false

[user]
INDEXED_VALUE = false

[http_session_id]
INDEXED_VALUE = false

[http_forward]
INDEXED_VALUE = false

[session_guid]
INDEXED_VALUE = false

[session_id]
INDEXED_VALUE = false

[datasource]
INDEXED_VALUE = false

[app_pool_name]
INDEXED_VALUE = false

[ip_address]
INDEXED_VALUE = false

[machine_name]
INDEXED_VALUE = false

[result]
INDEXED_VALUE = false

[message]
INDEXED_VALUE = false

[module]
INDEXED_VALUE = false

[class]
INDEXED_VALUE = false

[method]
INDEXED_VALUE = false

[source_file]
INDEXED_VALUE = false

[source_line]
INDEXED_VALUE = false

[severity]
INDEXED_VALUE = false

[error_id]
INDEXED_VALUE = false
 

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @jcgever,

I didn't check all your regex syntax but seems ok. Your transforms are wrong should be like below. 

[log_time]
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"
WRITE_META = true

[time_span]
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"
WRITE_META = true

[thread_id]
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"
WRITE_META = true

[user]
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"
WRITE_META = true

[http_session_id]
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"
WRITE_META = true

[http_forward]
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"
WRITE_META = true

[session_guid]
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"
WRITE_META = true

[session_id]
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"
WRITE_META = true

[datasource]
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"
WRITE_META = true

[app_pool_name]
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"
WRITE_META = true

[ip_address]
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"
WRITE_META = true

[machine_name]
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"
WRITE_META = true

[result]
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"
WRITE_META = true

[message]
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"
WRITE_META = true

[module]
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"
WRITE_META = true

[class]
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"
WRITE_META = true

[method]
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"
WRITE_META = true

[source_file]
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"
WRITE_META = true

[source_line]
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"
WRITE_META = true

[severity]
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"
WRITE_META = true

[error_id]
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"
WRITE_META = true

Also you must add these fields to fields.conf in your search head in order to be able to search;

[log_time]
INDEXED_VALUE = false

[time_span]
INDEXED_VALUE = false

[thread_id]
INDEXED_VALUE = false

[user]
INDEXED_VALUE = false

[http_session_id]
INDEXED_VALUE = false

[http_forward]
INDEXED_VALUE = false

[session_guid]
INDEXED_VALUE = false

[session_id]
INDEXED_VALUE = false

[datasource]
INDEXED_VALUE = false

[app_pool_name]
INDEXED_VALUE = false

[ip_address]
INDEXED_VALUE = false

[machine_name]
INDEXED_VALUE = false

[result]
INDEXED_VALUE = false

[message]
INDEXED_VALUE = false

[module]
INDEXED_VALUE = false

[class]
INDEXED_VALUE = false

[method]
INDEXED_VALUE = false

[source_file]
INDEXED_VALUE = false

[source_line]
INDEXED_VALUE = false

[severity]
INDEXED_VALUE = false

[error_id]
INDEXED_VALUE = false
 

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

jcgever
Explorer

Awesome that did it @scelikok ! Thank you so much for your help.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jcgever,

did you already tried with INDEXED_EXTRACTIONS = XML in props.conf?

for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Propsconf

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...