How to perform field extraction for key/value pairs with special character used as delimiter?

[SplunkDash]

Hello,

I have events with Key/Value pair assigned by "="Highlighted in Bold) and separated by special character "^".  Some Key/Value pair don't have values (Example Application Data). How would I perform field extraction for these events? 4 sample events are provided below. Any help will be highly appreciated. Thank you so much.

4 Sample Events

Application data=^Provider=TEST^ClientID=7acc2a917-309d-461b-806a-8b34ea6232aed8^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^SESSIONID=eZdfasRaMfTSSG2EDGUuT1UaYnWvk5rk=^AppID=OBA^TransD=4de099545e02-493s5-4720-9094-cef80cd71f7r3||2023-05-25T15:25:38.150Z||||||12.209.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=ayz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defba5c04b6c8481eca9d174d43cas^isass=https://api.test/oasidc^aud=[a255e650b9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49df545e02-493d5-4720-90df94-cef80cd71f7bv3||2023-05-25T15:25:38.097Z||||||10.208.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defse5c04b6c8481eca9d174d43c^isass=https://api.test/oidfdc^aud=[a255e650dfdb9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49sd545e02-493j5-4720-9ds094-cef80cd71fvv73||2023-05-25T15:25:38.094Z||||||12.208.9.173|

Application data=^Provider=eTEST^ID=9bsa5263e3-7423-4f01-a00f-4esse16e1693a8^givenName=MAAAMA^surName=SODA^dateofBirth=1968:10:20^address=92 Barca Boulevard^city=TAC^state=NH^zip=60000^email=mksds@gmail.com^phone=629-337-2349^alevel=D^SSIONID=pdffT2awi0gYTLbJo9kUtvosJsLnXNM=^AppID=OBA||2023-05-25T15:24:54.795Z||||||10.208.10.170|

[ITWhisperer]

| extract pairdelim="^" kvdelim="="

Extract works on the _raw field, so you may want to preserve the _raw field if you are trying to extract from another field

| rename _raw as temp
| rename field as _raw
| extract pairdelim="^" kvdelim="="
| rename _raw as field
| rename temp as _raw

[SplunkDash]

Hello @ITWhisperer,

Thank you so much for your quick response. But how I would implement it in my "In Line" or Transformations Field Extractions Or how I would incorporate it in props.conf/transforms.conf files. Any help will be appreciated. Thank you so much again.

[ITWhisperer]

tbh, I am not expert at this side of Splunk, but you could try this in your transforms.conf stanza

DELIMS = "^", "="

[isoutamo]

Hi

just like @ITWhisperer said, you should add props.conf (via file or GUI)

[<your sourcetype>]
REPORT-delims-by-carret = delims-by-carret

 then transforms.conf (this needs to add via file, as GUI wants also FIELD list).

[delims-by-carret]
DELIMS = "^","="
MV_ADD = 1

MV_ADD should be there if there could be MV value fields.

NOTE: Order of DELIMS character is important to get only wanted fieldnames. Unfortunately this seems to handle empty fields incorrectly (see e.g. data which contains all text until next space character. So you need to add additional handling for those fields if/when needed.

r. Ismo 

[SplunkDash]

@isoutamo 

I tried, but not sure why it's just extracted last 4 fields. Any thoughts?

[isoutamo · Accepted solution]

Probably you have some other settings which confusing that? With your examples, it work in my test environment, except those field name which contains space like "Application data". 

Can you try this on props.conf

[your sourcetype]
REPORT-test-delims-by-carret = extract-fields-by-carret

and in transforms.conf

[extract-fields-by-carret]
FORMAT = $1::$2
REGEX = ([^=]+)=([^\^]*)\^?

Unfortunately even this didn't extract correctly field "Application data" even it should. This works correctly on regex101.com.

I'm testing this on Splunk 9.0.4.1.

[SplunkDash]

Hello @isoutamo,

I know this is the best we can do for now. Thank you so much, truly appreciate your support.

[SplunkDash]

@isoutamo ,

Thank you so much for your quick response, truly appreciate it. Let me try it and get back to you. Thank you again.