Splunk Search

How would I extract field/value pairs from these sample events (2 sample events given below)?

SplunkDash
Motivator

Hello,

How would I extract field/value pairs from these sample events (2 sample events given below)? 

I can use like ......ID : (?P<ID>\w+)........but are there any good ways to get these key/value pairs. Thank you so much, would appreciate your support.

23:51:43.670 |LogMessage ID : sxntest ClientAddress : 10.207.68.172 Level : 6 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: Node ID: XP2SENTAtPCBUC1

23:51:43.789 |LogMessage ID : sxntest ClientAddress : 10.207.68.175 Level : 7 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: 09XV4R Node ID: XP2SENTXRTPCBUC

Labels (2)
Tags (1)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

Tricky to determine what is the field and what is the value. It seems that value follows : character, but field names appear to have spaces in them (LogMessage ID, Cluster ID, Node ID).  It's not clear if the value for CorrelationID is 'DetailsInfo' or nothing, as the data becomes a bit unclear. 

It looks like it's not possible to determine if the field has no value or not, e.g. see Cluster ID in the first row, which appears to have no value, compared with the second row, where it has a value.

Using the 'extract' command would normally allow you to get at these fields, but same issue applies in defining a consistent pattern. This SPL shows how one way to do this type of KV extraction, but it's a but clunky and does not work correctly because for Cluster ID in the first row it gets the word 'Node'.

| makeresults 
| eval x=split("23:51:43.670 |LogMessage ID : sxntest ClientAddress : 10.207.68.172 Level : 6 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: Node ID: XP2SENTAtPCBUC1###23:51:43.789 |LogMessage ID : sxntest ClientAddress : 10.207.68.175 Level : 7 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: 09XV4R Node ID: XP2SENTXRTPCBUC", "###")
| mvexpand x
| rename x as _raw
| rex "\|(?<fields>.*)"
| fields - _raw
| rex max_match=0 field=fields "(?<key>[^:]*): (?<value>[^ ]*)"
| eval key=trim(key)
| foreach 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [ eval _k=mvindex(key, <<FIELD>>), _v=mvindex(value, <<FIELD>>), {_k}=_v ]
| fields - _raw _k _v fields key value

If there is no general rule you can write regex for, then you're probably going to have to write a specific rex line to match field names + values as your example.

 

SplunkDash
Motivator

Hello,

Thank you so much. How could I use this in my props/transforms configuration files? Thank you again!

0 Karma

bowesmana
SplunkTrust
SplunkTrust

I'm not always 100% sure of the props/transforms, but in transforms.conf, you'd need

REGEX = .... define the regex here to capture key=value

and then the

FORMAT = $1::$2

to make the key=value as fields, however, unless you can get the relevant regex to guarantee the parsing, you'll end up with possible field mismatches

https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf

Here's a good description of setting it up

https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Exampleconfigurationsusingfieldtr...

 

0 Karma

chaker
Contributor

The problem is the key/value pairs are separated a major breaker (space). 

If you have control over the logging, removal of that space will result in automatic field extraction for the entire event, as : (colon) is a minor breaker.

https://docs.splunk.com/Documentation/SCS/current/Search/Eventsegmentationandsearching

Using a regular expression as you described is the best option here, unless you can transform the data to remove the space, however that in itself will be an expensive regex operation. You may as well just extract at search time using regex like the one you provided.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...