topic Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter. in Splunk Search

How to perform field extraction for key/value pairs with special character used as delimiter?

SplunkDash — Fri, 26 May 2023 12:42:33 GMT

Hello,

I have events with Key/Value pair assigned by "="Highlighted in Bold) and separated by special character "^". Some Key/Value pair don't have values (Example Application Data). How would I perform field extraction for these events? 4 sample events are provided below. Any help will be highly appreciated. Thank you so much.

4 Sample Events

Application data=^Provider=TEST^ClientID=7acc2a917-309d-461b-806a-8b34ea6232aed8^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^SESSIONID=eZdfasRaMfTSSG2EDGUuT1UaYnWvk5rk=^AppID=OBA^TransD=4de099545e02-493s5-4720-9094-cef80cd71f7r3||2023-05-25T15:25:38.150Z||||||12.209.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=ayz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defba5c04b6c8481eca9d174d43cas^isass=https://api.test/oasidc^aud=[a255e650b9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49df545e02-493d5-4720-90df94-cef80cd71f7bv3||2023-05-25T15:25:38.097Z||||||10.208.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defse5c04b6c8481eca9d174d43c^isass=https://api.test/oidfdc^aud=[a255e650dfdb9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49sd545e02-493j5-4720-9ds094-cef80cd71fvv73||2023-05-25T15:25:38.094Z||||||12.208.9.173|

Application data=^Provider=eTEST^ID=9bsa5263e3-7423-4f01-a00f-4esse16e1693a8^givenName=MAAAMA^surName=SODA^dateofBirth=1968:10:20^address=92 Barca Boulevard^city=TAC^state=NH^zip=60000^email=mksds@gmail.com^phone=629-337-2349^alevel=D^SSIONID=pdffT2awi0gYTLbJo9kUtvosJsLnXNM=^AppID=OBA||2023-05-25T15:24:54.795Z||||||10.208.10.170|

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

ITWhisperer — Fri, 26 May 2023 07:39:32 GMT

| extract pairdelim="^" kvdelim="="

Extract works on the _raw field, so you may want to preserve the _raw field if you are trying to extract from another field

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

SplunkDash — Fri, 26 May 2023 07:51:46 GMT

Hello @ITWhisperer,

Thank you so much for your quick response. But how I would implement it in my "In Line" or Transformations Field Extractions Or how I would incorporate it in props.conf/transforms.conf files. Any help will be appreciated. Thank you so much again.

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

ITWhisperer — Fri, 26 May 2023 08:00:12 GMT

tbh, I am not expert at this side of Splunk, but you could try this in your transforms.conf stanza

DELIMS = "^", "="

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

isoutamo — Fri, 26 May 2023 09:13:05 GMT

just like @ITWhisperer said, you should add props.conf (via file or GUI)

[<your sourcetype>] REPORT-delims-by-carret = delims-by-carret

then transforms.conf (this needs to add via file, as GUI wants also FIELD list).

[delims-by-carret] DELIMS = "^","=" MV_ADD = 1

MV_ADD should be there if there could be MV value fields.

NOTE: Order of DELIMS character is important to get only wanted fieldnames. Unfortunately this seems to handle empty fields incorrectly (see e.g. data which contains all text until next space character. So you need to add additional handling for those fields if/when needed.

r. Ismo

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

SplunkDash — Fri, 26 May 2023 09:11:28 GMT

@isoutamo ,

Thank you so much for your quick response, truly appreciate it. Let me try it and get back to you. Thank you again.

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

SplunkDash — Fri, 26 May 2023 10:27:56 GMT

@isoutamo

I tried, but not sure why it's just extracted last 4 fields. Any thoughts?

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

isoutamo — Fri, 26 May 2023 12:08:07 GMT

Probably you have some other settings which confusing that? With your examples, it work in my test environment, except those field name which contains space like "Application data".

Can you try this on props.conf

[your sourcetype] REPORT-test-delims-by-carret = extract-fields-by-carret

and in transforms.conf

[extract-fields-by-carret] FORMAT = $1::$2 REGEX = ([^=]+)=([^\^]*)\^?

Unfortunately even this didn't extract correctly field "Application data" even it should. This works correctly on regex101.com.

I'm testing this on Splunk 9.0.4.1.

Re: Field Extraction -Key/ Value Pairs with Special Character used as Delimiter.

SplunkDash — Fri, 26 May 2023 12:37:20 GMT

Hello @isoutamo,

I know this is the best we can do for now. Thank you so much, truly appreciate your support.