Dynamic multiple field and value extraction

ronpestler1 — Wed, 30 Sep 2020 03:03:06 GMT

Hello together,

i use splunk the version 7.2.4.2 and had the following issue by creating a dynamic field exctration.
This is a Event Example:

##### BEGIN EVENT #####
Mode
UPDATE
----- END OF FIELD -----
Identifier
probe_connection_event:A1:servername123:65:PROBE::
----- END OF FIELD -----
Serial
1234567891
----- END OF FIELD -----
Node
servername123
----- END OF FIELD -----
NodeAlias
----- END OF FIELD -----
Manager
Bus
Self Monitoring @A1
----- END OF FIELD -----
Agent
SelfMonitoring
----- END OF FIELD -----
AlertGroup
ConnectionStatus
##### END EVENT #####

So the field construction is the following one:

<field name> [single line] 
<field value> [multi line or empty] 
----- END OF FIELD ----- [OR] ##### END EVENT #####

For that i created with regex101 the following regex:

(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)

When i test the regex against a small set it extracts me the first correct:

/opt/splunk/bin/splunk cmd pcregextest mregex='(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'  test_str="##### BEGIN EVENT #####
> Mode
> DELETE
> ----- END OF FIELD -----
> ServerSerial
> 86919394
> ----- END OF FIELD -----
> ServerName
> AA1
> ##### END EVENT #####"
Original Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Expanded Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Regex compiled successfully. Capture group count = 7. Named capturing groups = 0.
SUCCESS - match against: '##### BEGIN EVENT #####
Mode
DELETE
----- END OF FIELD -----
ServerSerial
86919394
----- END OF FIELD -----
ServerName
AA1
##### END EVENT #####'

#### Capturing group data #####
Group |            Name | Value
--------------------------------------
    1 |                 | #####
    2 |                 |

    3 |                 | Mode
    4 |                 |

    5 |                 | DELETE
    6 |                 |

    7 |                 | ----- END OF FIELD

My Props.conf is the following one:

[custom_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
TRUNCATE=512
category=CUSTOM
disabled=false
pulldown_type=true
MAX_EVENTS=4096
BREAK_ONLY_BEFORE=\#\#\#\#\ BEGIN\ EVENT\ \#\#\#\#
REPORT-omnifields=omnifieldsExtract

And my Transforms.conf:

[omnifieldsExtract]
REGEX = (?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)
REPEAT_MATCH = true
FORMAT = $3::$5

The transforms and props configuration are stored on the search head cluster.

So my opinion is now for the event example that i get the following fields:

Field Name: Mode
Value: UPDATE
Field Name: Identifier
Value: probe_connection_event:A1:servername123:65:PROBE::
Field Name: Serial
Value: 1234567891
Field Name: Node
Value: servername123
Field Name: NodeAlias
Value:
Field Name: Manager
Value: "Bus
Self Monitoring @A1"
Field Name: Agent
Value: SelfMonitoring
Field Name: AlertGroup
Value: ConnectionStatus

But actually i get no single field.
I didnt know what i am missing or maked false. Had anyone a idea?
Thanks for your help 😄

Re: Dynamic multiple field and value extraction

richgalloway — Thu, 21 Nov 2019 12:59:36 GMT

Put the props.conf and transforms.conf files on your indexers and restart them.

Re: Dynamic multiple field and value extraction

ronpestler1 — Thu, 21 Nov 2019 16:21:19 GMT

okay, i putted both files now on indexers, before i had only the props.conf without the REPORT line.
But its still the same, no field exctration happen.

topic Dynamic multiple field and value extraction in Splunk Search

Dynamic multiple field and value extraction

Re: Dynamic multiple field and value extraction

Re: Dynamic multiple field and value extraction