Getting Data In

Help in parsing logs- How to cut out or blacklist the first portion and only grabbing the col name?

jcgever
Explorer

Been trying to work out the regex for the props and transforms for onbase logging and I keep hitting a brick wall. Here is a sample log of what I have been working on:

<?xml version="1.0" encoding="utf-8"?>
<diagnosticsLog type="error-profile" startDate="07/28/2022 01:10:20">
<!--Build 18.0.1.42-->
<columns>
<column friendlyName="time" name="time" />
<column friendlyName="Result" name="Result" />
<column friendlyName="Module" name="Module" />
<column friendlyName="Class" name="Class" />
<column friendlyName="SourceFile" name="SourceFile" />
<column friendlyName="Method" name="Method" />
<column friendlyName="SourceLine" name="SourceLine" />
<column friendlyName="Severity" name="Severity" />
<column friendlyName="MachineName" name="MachineName" />
<column friendlyName="IpAddress" name="IpAddress" />
<column friendlyName="ErrorId" name="ErrorId" />
<column friendlyName="ProcessID" name="ProcessID" />
<column friendlyName="ThreadID" name="ThreadID" />
<column friendlyName="TimeSpan" name="TimeSpan" />
<column friendlyName="User" name="User" />
<column friendlyName="HTTPSessionID" name="HTTPSessionID" />
<column friendlyName="HTTPForward" name="HTTPForward" />
<column friendlyName="SessionID" name="SessionID" />
<column friendlyName="SessionGUID" name="SessionGUID" />
<column friendlyName="Datasource" name="Datasource" />
<column friendlyName="Sequence" name="Sequence" />
<column friendlyName="LocalSequence" name="LocalSequence" />
<column friendlyName="Message" name="Message" />
<column friendlyName="AppPoolName" name="AppPoolName" />
</columns>
<rows>
<row>
<col name="time">07/28/2022 01:10:20</col>
<col name="TimeSpan">N/A</col>
<col name="ThreadID">0x0000000A</col>
<col name="User"></col>
<col name="HTTPSessionID"></col>
<col name="HTTPForward"></col>
<col name="SessionGUID"></col>
<col name="SessionID">0</col>
<col name="Datasource">OnBaseQA</col>
<col name="AppPoolName"></col>
<col name="IpAddress"></col>
<col name="MachineName"></col>
<col name="Result">0x00000000</col>
<col name="Message">FileLoadException: File [D:\Program Files\Hyland\Services\Distribution\System.Runtime.CompilerServices.Unsafe.dll] Message [Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=4.0.3.0, Culture=neutral, PublicKeyToken=abcd1234' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x901234)]</col>
<col name="Module">Hyland.Core</col>
<col name="Class">AssemblyRegistration</col>
<col name="Method">ReadAssemblyXml</col>
<col name="SourceFile"></col>
<col name="SourceLine">0</col>
<col name="Severity">Error</col>
<col name="ErrorId"></col>
</row>

I'm trying to cut out or blacklist the first portion with the column naming and only grabbing the col name in between the "" with the corresponding data. I'm very new to manipulating data like this and am just starting to understand regex. These are my attempts on props and transforms:

 

Transforms:

[log_time]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"

[time_span]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"

[thread_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"

[user]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"

[http_session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"

[http_forward]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"

[session_guid]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"

[session_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"

[datasource]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"

[app_pool_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"

[ip_address]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"

[machine_name]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"

[result]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"

[message]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"

[module]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"

[class]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"

[method]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"

[source_file]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"

[source_line]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"

[severity]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"

[error_id]
DEST_KEY = MetaData:Sourcetype
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"

 

Props:
[error_profile]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
TRANSFORMS-set = log_time, time_span, thread_id, user, http_session_id, http_forward, session_guid, session_id, datasource, app_pool_name, ip_address, machine_name, result, message, module, class, method, source_file, source_line, severity, error_id

Labels (2)
Tags (2)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @jcgever,

I didn't check all your regex syntax but seems ok. Your transforms are wrong should be like below. 

[log_time]
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"
WRITE_META = true

[time_span]
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"
WRITE_META = true

[thread_id]
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"
WRITE_META = true

[user]
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"
WRITE_META = true

[http_session_id]
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"
WRITE_META = true

[http_forward]
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"
WRITE_META = true

[session_guid]
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"
WRITE_META = true

[session_id]
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"
WRITE_META = true

[datasource]
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"
WRITE_META = true

[app_pool_name]
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"
WRITE_META = true

[ip_address]
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"
WRITE_META = true

[machine_name]
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"
WRITE_META = true

[result]
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"
WRITE_META = true

[message]
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"
WRITE_META = true

[module]
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"
WRITE_META = true

[class]
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"
WRITE_META = true

[method]
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"
WRITE_META = true

[source_file]
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"
WRITE_META = true

[source_line]
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"
WRITE_META = true

[severity]
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"
WRITE_META = true

[error_id]
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"
WRITE_META = true

Also you must add these fields to fields.conf in your search head in order to be able to search;

[log_time]
INDEXED_VALUE = false

[time_span]
INDEXED_VALUE = false

[thread_id]
INDEXED_VALUE = false

[user]
INDEXED_VALUE = false

[http_session_id]
INDEXED_VALUE = false

[http_forward]
INDEXED_VALUE = false

[session_guid]
INDEXED_VALUE = false

[session_id]
INDEXED_VALUE = false

[datasource]
INDEXED_VALUE = false

[app_pool_name]
INDEXED_VALUE = false

[ip_address]
INDEXED_VALUE = false

[machine_name]
INDEXED_VALUE = false

[result]
INDEXED_VALUE = false

[message]
INDEXED_VALUE = false

[module]
INDEXED_VALUE = false

[class]
INDEXED_VALUE = false

[method]
INDEXED_VALUE = false

[source_file]
INDEXED_VALUE = false

[source_line]
INDEXED_VALUE = false

[severity]
INDEXED_VALUE = false

[error_id]
INDEXED_VALUE = false
 

 

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @jcgever,

I didn't check all your regex syntax but seems ok. Your transforms are wrong should be like below. 

[log_time]
REGEX = <col name="time">(.*)<\/col>
FORMAT = time::"$1"
WRITE_META = true

[time_span]
REGEX = <col name="TimeSpan">(.*)<\/col>
FORMAT = time_span::"$1"
WRITE_META = true

[thread_id]
REGEX = <col name="ThreadID">(.*)<\/col>
FORMAT = thread_id::"$1"
WRITE_META = true

[user]
REGEX = <col name="User">(.*)<\/col>
FORMAT = user::"$1"
WRITE_META = true

[http_session_id]
REGEX = <col name="HTTPSessionID">(.*)<\/col>
FORMAT = http_session_id::"$1"
WRITE_META = true

[http_forward]
REGEX = <col name="HTTPForward">(.*)<\/col>
FORMAT = http_forward::"$1"
WRITE_META = true

[session_guid]
REGEX = <col name="SessionGUID">(.*)<\/col>
FORMAT = session_guid::"$1"
WRITE_META = true

[session_id]
REGEX = <col name="SessionID">(.*)<\/col>
FORMAT = session_id::"$1"
WRITE_META = true

[datasource]
REGEX = <col name="Datasource">(.*)<\/col>
FORMAT = datasource::"$1"
WRITE_META = true

[app_pool_name]
REGEX = <col name="AppPoolName">(.*)<\/col>
FORMAT = app_pool_name::"$1"
WRITE_META = true

[ip_address]
REGEX = <col name="IpAddress">(.*)<\/col>
FORMAT = ip_address::"$1"
WRITE_META = true

[machine_name]
REGEX = <col name="MachineName">(.*)<\/col>
FORMAT = machine_name::"$1"
WRITE_META = true

[result]
REGEX = <col name="Result">(.*)<\/col>
FORMAT = result::"$1"
WRITE_META = true

[message]
REGEX = <col name="Message">(.*)<\/col>
FORMAT = message::"$1"
WRITE_META = true

[module]
REGEX = <col name="Module">(.*)<\/col>
FORMAT = module::"$1"
WRITE_META = true

[class]
REGEX = <col name="Class">(.*)<\/col>
FORMAT = class::"$1"
WRITE_META = true

[method]
REGEX = <col name="Method">(.*)<\/col>
FORMAT = method::"$1"
WRITE_META = true

[source_file]
REGEX = <col name="SourceFile">(.*)<\/col>
FORMAT = source_file::"$1"
WRITE_META = true

[source_line]
REGEX = <col name="SourceLine">(.*)<\/col>
FORMAT = source_line::"$1"
WRITE_META = true

[severity]
REGEX = <col name="Severity">(.*)<\/col>
FORMAT = severity::"$1"
WRITE_META = true

[error_id]
REGEX = <col name="ErrorId">(.*)<\/col>
FORMAT = error_id::"$1"
WRITE_META = true

Also you must add these fields to fields.conf in your search head in order to be able to search;

[log_time]
INDEXED_VALUE = false

[time_span]
INDEXED_VALUE = false

[thread_id]
INDEXED_VALUE = false

[user]
INDEXED_VALUE = false

[http_session_id]
INDEXED_VALUE = false

[http_forward]
INDEXED_VALUE = false

[session_guid]
INDEXED_VALUE = false

[session_id]
INDEXED_VALUE = false

[datasource]
INDEXED_VALUE = false

[app_pool_name]
INDEXED_VALUE = false

[ip_address]
INDEXED_VALUE = false

[machine_name]
INDEXED_VALUE = false

[result]
INDEXED_VALUE = false

[message]
INDEXED_VALUE = false

[module]
INDEXED_VALUE = false

[class]
INDEXED_VALUE = false

[method]
INDEXED_VALUE = false

[source_file]
INDEXED_VALUE = false

[source_line]
INDEXED_VALUE = false

[severity]
INDEXED_VALUE = false

[error_id]
INDEXED_VALUE = false
 

 

If this reply helps you an upvote is appreciated.

jcgever
Explorer

Awesome that did it @scelikok ! Thank you so much for your help.

0 Karma

gcusello
Esteemed Legend

Hi @jcgever,

did you already tried with INDEXED_EXTRACTIONS = XML in props.conf?

for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Propsconf

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...