Help in parsing logs- How to cut out or blacklist the first portion and only grabbing the col name?

[jcgever]

Been trying to work out the regex for the props and transforms for onbase logging and I keep hitting a brick wall. Here is a sample log of what I have been working on:

<?xml version="1.0" encoding="utf-8"?>
<diagnosticsLog type="error-profile" startDate="07/28/2022 01:10:20">
<!--Build 18.0.1.42-->
<columns>
<column friendlyName="time" name="time" />
<column friendlyName="Result" name="Result" />
<column friendlyName="Module" name="Module" />
<column friendlyName="Class" name="Class" />
<column friendlyName="SourceFile" name="SourceFile" />
<column friendlyName="Method" name="Method" />
<column friendlyName="SourceLine" name="SourceLine" />
<column friendlyName="Severity" name="Severity" />
<column friendlyName="MachineName" name="MachineName" />
<column friendlyName="IpAddress" name="IpAddress" />
<column friendlyName="ErrorId" name="ErrorId" />
<column friendlyName="ProcessID" name="ProcessID" />
<column friendlyName="ThreadID" name="ThreadID" />
<column friendlyName="TimeSpan" name="TimeSpan" />
<column friendlyName="User" name="User" />
<column friendlyName="HTTPSessionID" name="HTTPSessionID" />
<column friendlyName="HTTPForward" name="HTTPForward" />
<column friendlyName="SessionID" name="SessionID" />
<column friendlyName="SessionGUID" name="SessionGUID" />
<column friendlyName="Datasource" name="Datasource" />
<column friendlyName="Sequence" name="Sequence" />
<column friendlyName="LocalSequence" name="LocalSequence" />
<column friendlyName="Message" name="Message" />
<column friendlyName="AppPoolName" name="AppPoolName" />
</columns>
<rows>
<row>
<col name="time">07/28/2022 01:10:20</col>
<col name="TimeSpan">N/A</col>
<col name="ThreadID">0x0000000A</col>
<col name="User"></col>
<col name="HTTPSessionID"></col>
<col name="HTTPForward"></col>
<col name="SessionGUID"></col>
<col name="SessionID">0</col>
<col name="Datasource">OnBaseQA</col>
<col name="AppPoolName"></col>
<col name="IpAddress"></col>
<col name="MachineName"></col>
<col name="Result">0x00000000</col>
<col name="Message">FileLoadException: File [D:\Program Files\Hyland\Services\Distribution\System.Runtime.CompilerServices.Unsafe.dll] Message [Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=4.0.3.0, Culture=neutral, PublicKeyToken=abcd1234' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x901234)]</col>
<col name="Module">Hyland.Core</col>
<col name="Class">AssemblyRegistration</col>
<col name="Method">ReadAssemblyXml</col>
<col name="SourceFile"></col>
<col name="SourceLine">0</col>
<col name="Severity">Error</col>
<col name="ErrorId"></col>
</row>

I'm trying to cut out or blacklist the first portion with the column naming and only grabbing the col name in between the "" with the corresponding data. I'm very new to manipulating data like this and am just starting to understand regex. These are my attempts on props and transforms:

Transforms:

[log_time]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"

[time_span]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"

[thread_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"

[user]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"

[http_session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"

[http_forward]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"

[session_guid]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"

[session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"

[datasource]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"

[app_pool_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"

[ip_address]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"

[machine_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"

[result]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"

[message]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"

[module]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"

[class]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"

[method]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"

[source_file]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"

[source_line]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"

[severity]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"

[error_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"

Props:
[error_profile]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
TRANSFORMS-set = log_time, time_span, thread_id, user, http_session_id, http_forward, session_guid, session_id, datasource, app_pool_name, ip_address, machine_name, result, message, module, class, method, source_file, source_line, severity, error_id

[scelikok · Accepted solution]

Hi @jcgever,

I didn't check all your regex syntax but seems ok. Your transforms are wrong should be like below. 

[log_time]
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"
WRITE_META = true

[time_span]
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"
WRITE_META = true

[thread_id]
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"
WRITE_META = true

[user]
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"
WRITE_META = true

[http_session_id]
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"
WRITE_META = true

[http_forward]
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"
WRITE_META = true

[session_guid]
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"
WRITE_META = true

[session_id]
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"
WRITE_META = true

[datasource]
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"
WRITE_META = true

[app_pool_name]
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"
WRITE_META = true

[ip_address]
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"
WRITE_META = true

[machine_name]
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"
WRITE_META = true

[result]
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"
WRITE_META = true

[message]
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"
WRITE_META = true

[module]
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"
WRITE_META = true

[class]
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"
WRITE_META = true

[method]
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"
WRITE_META = true

[source_file]
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"
WRITE_META = true

[source_line]
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"
WRITE_META = true

[severity]
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"
WRITE_META = true

[error_id]
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"
WRITE_META = true

Also you must add these fields to fields.conf in your search head in order to be able to search;

[log_time]
INDEXED_VALUE = false

[time_span]
INDEXED_VALUE = false

[thread_id]
INDEXED_VALUE = false

[user]
INDEXED_VALUE = false

[http_session_id]
INDEXED_VALUE = false

[http_forward]
INDEXED_VALUE = false

[session_guid]
INDEXED_VALUE = false

[session_id]
INDEXED_VALUE = false

[datasource]
INDEXED_VALUE = false

[app_pool_name]
INDEXED_VALUE = false

[ip_address]
INDEXED_VALUE = false

[machine_name]
INDEXED_VALUE = false

[result]
INDEXED_VALUE = false

[message]
INDEXED_VALUE = false

[module]
INDEXED_VALUE = false

[class]
INDEXED_VALUE = false

[method]
INDEXED_VALUE = false

[source_file]
INDEXED_VALUE = false

[source_line]
INDEXED_VALUE = false

[severity]
INDEXED_VALUE = false

[error_id]
INDEXED_VALUE = false
 

[jcgever]

Awesome that did it @scelikok ! Thank you so much for your help.

[gcusello]

Hi @jcgever,

did you already tried with INDEXED_EXTRACTIONS = XML in props.conf?

for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Propsconf

Ciao.

Giuseppe