topic Re: Extract field value pairs in Splunk Search

How would I extract field/value pairs from these sample events (2 sample events given below)?

SplunkDash — Thu, 25 Aug 2022 00:25:43 GMT

Hello,

How would I extract field/value pairs from these sample events (2 sample events given below)?

I can use like ......ID : (?P<ID>\w+)........but are there any good ways to get these key/value pairs. Thank you so much, would appreciate your support.

23:51:43.670 |LogMessage ID : sxntest ClientAddress : 10.207.68.172 Level : 6 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: Node ID: XP2SENTAtPCBUC1

23:51:43.789 |LogMessage ID : sxntest ClientAddress : 10.207.68.175 Level : 7 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: 09XV4R Node ID: XP2SENTXRTPCBUC

Re: Extract field value pairs

chaker — Wed, 24 Aug 2022 23:58:34 GMT

The problem is the key/value pairs are separated a major breaker (space).

If you have control over the logging, removal of that space will result in automatic field extraction for the entire event, as : (colon) is a minor breaker.

https://docs.splunk.com/Documentation/SCS/current/Search/Eventsegmentationandsearching

Using a regular expression as you described is the best option here, unless you can transform the data to remove the space, however that in itself will be an expensive regex operation. You may as well just extract at search time using regex like the one you provided.

Re: Extract field value pairs

bowesmana — Thu, 25 Aug 2022 00:16:57 GMT

Tricky to determine what is the field and what is the value. It seems that value follows : character, but field names appear to have spaces in them (LogMessage ID, Cluster ID, Node ID). It's not clear if the value for CorrelationID is 'DetailsInfo' or nothing, as the data becomes a bit unclear.

It looks like it's not possible to determine if the field has no value or not, e.g. see Cluster ID in the first row, which appears to have no value, compared with the second row, where it has a value.

Using the 'extract' command would normally allow you to get at these fields, but same issue applies in defining a consistent pattern. This SPL shows how one way to do this type of KV extraction, but it's a but clunky and does not work correctly because for Cluster ID in the first row it gets the word 'Node'.

| makeresults | eval x=split("23:51:43.670 |LogMessage ID : sxntest ClientAddress : 10.207.68.172 Level : 6 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: Node ID: XP2SENTAtPCBUC1###23:51:43.789 |LogMessage ID : sxntest ClientAddress : 10.207.68.175 Level : 7 EventType : UserLogging Resource: RESTIP EventStatus : Success CEvent : No Category : TestEvent ComID : VMREST CorrelationID : DetailsInfo : Login App ID: DSTest Cluster ID: 09XV4R Node ID: XP2SENTXRTPCBUC", "###") | mvexpand x | rename x as _raw | rex "\|(?<fields>.*)" | fields - _raw | rex max_match=0 field=fields "(?<key>[^:]*): (?<value>[^ ]*)" | eval key=trim(key) | foreach 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [ eval _k=mvindex(key, <<FIELD>>), _v=mvindex(value, <<FIELD>>), {_k}=_v ] | fields - _raw _k _v fields key value

If there is no general rule you can write regex for, then you're probably going to have to write a specific rex line to match field names + values as your example.

Re: Extract field value pairs

SplunkDash — Thu, 25 Aug 2022 00:47:03 GMT

Hello,

Thank you so much. How could I use this in my props/transforms configuration files? Thank you again!

Re: Extract field value pairs

bowesmana — Thu, 25 Aug 2022 01:03:02 GMT

I'm not always 100% sure of the props/transforms, but in transforms.conf, you'd need

REGEX = .... define the regex here to capture key=value

and then the

FORMAT = $1::$2

to make the key=value as fields, however, unless you can get the relevant regex to guarantee the parsing, you'll end up with possible field mismatches

https://docs.splunk.com/Documentation/Splunk/latest/Admin/transformsconf

Here's a good description of setting it up

https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Exampleconfigurationsusingfieldtransforms