Splunk Search

Dynamic multiple field and value extraction

ronpestler1
Explorer

Hello together,

i use splunk the version 7.2.4.2 and had the following issue by creating a dynamic field exctration.
This is a Event Example:

##### BEGIN EVENT #####
Mode
UPDATE
----- END OF FIELD -----
Identifier
probe_connection_event:A1:servername123:65:PROBE::
----- END OF FIELD -----
Serial
1234567891
----- END OF FIELD -----
Node
servername123
----- END OF FIELD -----
NodeAlias
----- END OF FIELD -----
Manager
Bus
Self Monitoring @A1
----- END OF FIELD -----
Agent
SelfMonitoring
----- END OF FIELD -----
AlertGroup
ConnectionStatus
##### END EVENT #####

So the field construction is the following one:

<field name> [single line] 
<field value> [multi line or empty] 
----- END OF FIELD ----- [OR] ##### END EVENT #####

For that i created with regex101 the following regex:

(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)

When i test the regex against a small set it extracts me the first correct:

/opt/splunk/bin/splunk cmd pcregextest mregex='(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'  test_str="##### BEGIN EVENT #####
> Mode
> DELETE
> ----- END OF FIELD -----
> ServerSerial
> 86919394
> ----- END OF FIELD -----
> ServerName
> AA1
> ##### END EVENT #####"
Original Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Expanded Pattern: '(?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)'
Regex compiled successfully. Capture group count = 7. Named capturing groups = 0.
SUCCESS - match against: '##### BEGIN EVENT #####
Mode
DELETE
----- END OF FIELD -----
ServerSerial
86919394
----- END OF FIELD -----
ServerName
AA1
##### END EVENT #####'

#### Capturing group data #####
Group |            Name | Value
--------------------------------------
    1 |                 | #####
    2 |                 |

    3 |                 | Mode
    4 |                 |

    5 |                 | DELETE
    6 |                 |

    7 |                 | ----- END OF FIELD

My Props.conf is the following one:

[custom_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
TRUNCATE=512
category=CUSTOM
disabled=false
pulldown_type=true
MAX_EVENTS=4096
BREAK_ONLY_BEFORE=\#\#\#\#\ BEGIN\ EVENT\ \#\#\#\#
REPORT-omnifields=omnifieldsExtract

And my Transforms.conf:

[omnifieldsExtract]
REGEX = (?m)(\#\#\#\#\#|\-\-\-\-\-)$(\r\n|\r|\n)^(.*)$(\r\n|\r|\n)?^([\s\S]*?)(\r\n|\r|\n)?(\#\#\#\#\#|\-\-\-\-\-\ END\ OF\ FIELD)
REPEAT_MATCH = true
FORMAT = $3::$5

The transforms and props configuration are stored on the search head cluster.

So my opinion is now for the event example that i get the following fields:

Field Name: Mode
Value: UPDATE
Field Name: Identifier
Value: probe_connection_event:A1:servername123:65:PROBE::
Field Name: Serial
Value: 1234567891
Field Name: Node
Value: servername123
Field Name: NodeAlias
Value:
Field Name: Manager
Value: "Bus
Self Monitoring @A1"
Field Name: Agent
Value: SelfMonitoring
Field Name: AlertGroup
Value: ConnectionStatus

But actually i get no single field.
I didnt know what i am missing or maked false. Had anyone a idea?
Thanks for your help 😄

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Put the props.conf and transforms.conf files on your indexers and restart them.

---
If this reply helps you, Karma would be appreciated.
0 Karma

ronpestler1
Explorer

okay, i putted both files now on indexers, before i had only the props.conf without the REPORT line.
But its still the same, no field exctration happen.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...