Dynamic multiple field and value extraction

[ronpestler1]

Hello together,

i use splunk the version 7.2.4.2 and had the following issue by creating a dynamic field exctration.
This is a Event Example:

##### BEGIN EVENT #####
Mode
UPDATE
----- END OF FIELD -----
Identifier
probe_connection_event:A1:servername123:65:PROBE::
----- END OF FIELD -----
Serial
1234567891
----- END OF FIELD -----
Node
servername123
----- END OF FIELD -----
NodeAlias
----- END OF FIELD -----
Manager
Bus
Self Monitoring @A1
----- END OF FIELD -----
Agent
SelfMonitoring
----- END OF FIELD -----
AlertGroup
ConnectionStatus
##### END EVENT #####

So the field construction is the following one:

<field name> [single line] 
<field value> [multi line or empty] 
----- END OF FIELD ----- [OR] ##### END EVENT #####

For that i created with regex101 the following regex:

(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)

When i test the regex against a small set it extracts me the first correct:

/opt/splunk/bin/splunk cmd pcregextest mregex='(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'  test_str="##### BEGIN EVENT #####
> Mode
> DELETE
> ----- END OF FIELD -----
> ServerSerial
> 86919394
> ----- END OF FIELD -----
> ServerName
> AA1
> ##### END EVENT #####"
Original Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Expanded Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Regex compiled successfully. Capture group count = 7. Named capturing groups = 0.
SUCCESS - match against: '##### BEGIN EVENT #####
Mode
DELETE
----- END OF FIELD -----
ServerSerial
86919394
----- END OF FIELD -----
ServerName
AA1
##### END EVENT #####'

#### Capturing group data #####
Group |            Name | Value
--------------------------------------
    1 |                 | #####
    2 |                 |

    3 |                 | Mode
    4 |                 |

    5 |                 | DELETE
    6 |                 |

    7 |                 | ----- END OF FIELD

My Props.conf is the following one:

[custom_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
TRUNCATE=512
category=CUSTOM
disabled=false
pulldown_type=true
MAX_EVENTS=4096
BREAK_ONLY_BEFORE=\#\#\#\#\ BEGIN\ EVENT\ \#\#\#\#
REPORT-omnifields=omnifieldsExtract

And my Transforms.conf:

[omnifieldsExtract]
REGEX = (?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)
REPEAT_MATCH = true
FORMAT = $3::$5

The transforms and props configuration are stored on the search head cluster.

So my opinion is now for the event example that i get the following fields:

Field Name: Mode
Value: UPDATE
Field Name: Identifier
Value: probe_connection_event:A1:servername123:65:PROBE::
Field Name: Serial
Value: 1234567891
Field Name: Node
Value: servername123
Field Name: NodeAlias
Value:
Field Name: Manager
Value: "Bus
Self Monitoring @A1"
Field Name: Agent
Value: SelfMonitoring
Field Name: AlertGroup
Value: ConnectionStatus

But actually i get no single field.
I didnt know what i am missing or maked false. Had anyone a idea?
Thanks for your help 😄

[richgalloway]

Put the props.conf and transforms.conf files on your indexers and restart them.

[ronpestler1]

okay, i putted both files now on indexers, before i had only the props.conf without the REPORT line.
But its still the same, no field exctration happen.