Splunk Search

How to perform field extraction for key/value pairs with special character used as delimiter?

SplunkDash
Motivator

Hello,

I have events with Key/Value pair assigned by "="Highlighted in Bold) and separated by special character "^".  Some Key/Value pair don't have values (Example Application Data). How would I perform field extraction for these events? 4 sample events are provided below. Any help will be highly appreciated. Thank you so much.

4 Sample Events

Application data=^Provider=TEST^ClientID=7acc2a917-309d-461b-806a-8b34ea6232aed8^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^SESSIONID=eZdfasRaMfTSSG2EDGUuT1UaYnWvk5rk=^AppID=OBA^TransD=4de099545e02-493s5-4720-9094-cef80cd71f7r3||2023-05-25T15:25:38.150Z||||||12.209.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=ayz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defba5c04b6c8481eca9d174d43cas^isass=https://api.test/oasidc^aud=[a255e650b9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49df545e02-493d5-4720-90df94-cef80cd71f7bv3||2023-05-25T15:25:38.097Z||||||10.208.9.173|

Application data=^Provider=TEST^givenName=XYZ^surName=ABC^dateofBirth=1970-10-05^address=1940n Xaybas St Apt 1^city=ABC^state=NY^zip=50000^email=xyz@gmail.com^phone=10974173700^alevel=IAB7^sub=20216defse5c04b6c8481eca9d174d43c^isass=https://api.test/oidfdc^aud=[a255e650dfdb9a8194b0264468854b57b41]^exp=Thu May 25 21:25:37 EDT 2023^iat=Thu May 25 11:25:37 EDT 2023^AppID=OBA^TransID=49sd545e02-493j5-4720-9ds094-cef80cd71fvv73||2023-05-25T15:25:38.094Z||||||12.208.9.173|

Application data=^Provider=eTEST^ID=9bsa5263e3-7423-4f01-a00f-4esse16e1693a8^givenName=MAAAMA^surName=SODA^dateofBirth=1968:10:20^address=92 Barca Boulevard^city=TAC^state=NH^zip=60000^email=mksds@gmail.com^phone=629-337-2349^alevel=D^SSIONID=pdffT2awi0gYTLbJo9kUtvosJsLnXNM=^AppID=OBA||2023-05-25T15:24:54.795Z||||||10.208.10.170|

Labels (1)
Tags (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Probably you have some other settings which confusing that? With your examples, it work in my test environment, except those field name which contains space like "Application data". 

Can you try this on props.conf

 

[your sourcetype]
REPORT-test-delims-by-carret = extract-fields-by-carret

 

and in transforms.conf

 

[extract-fields-by-carret]
FORMAT = $1::$2
REGEX = ([^=]+)=([^\^]*)\^?

 

Unfortunately even this didn't extract correctly field "Application data" even it should. This works correctly on regex101.com.

I'm testing this on Splunk 9.0.4.1.

isoutamo_0-1685102877611.png

 

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| extract pairdelim="^" kvdelim="="

Extract works on the _raw field, so you may want to preserve the _raw field if you are trying to extract from another field

| rename _raw as temp
| rename field as _raw
| extract pairdelim="^" kvdelim="="
| rename _raw as field
| rename temp as _raw

SplunkDash
Motivator

Hello @ITWhisperer,

Thank you so much for your quick response. But how I would implement it in my "In Line" or Transformations Field Extractions Or how I would incorporate it in props.conf/transforms.conf files. Any help will be appreciated. Thank you so much again.

 

Tags (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

tbh, I am not expert at this side of Splunk, but you could try this in your transforms.conf stanza

DELIMS = "^", "="

isoutamo
SplunkTrust
SplunkTrust

Hi

just like @ITWhisperer said, you should add props.conf (via file or GUI)

 

[<your sourcetype>]
REPORT-delims-by-carret = delims-by-carret

 

 then transforms.conf (this needs to add via file, as GUI wants also FIELD list).

 

[delims-by-carret]
DELIMS = "^","="
MV_ADD = 1

 

MV_ADD should be there if there could be MV value fields.

NOTE: Order of DELIMS character is important to get only wanted fieldnames. Unfortunately this seems to handle empty fields incorrectly (see e.g. data which contains all text until next space character. So you need to add additional handling for those fields if/when needed.

r. Ismo 

SplunkDash
Motivator

@isoutamo 

I tried, but not sure why it's just extracted last 4 fields. Any thoughts?

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Probably you have some other settings which confusing that? With your examples, it work in my test environment, except those field name which contains space like "Application data". 

Can you try this on props.conf

 

[your sourcetype]
REPORT-test-delims-by-carret = extract-fields-by-carret

 

and in transforms.conf

 

[extract-fields-by-carret]
FORMAT = $1::$2
REGEX = ([^=]+)=([^\^]*)\^?

 

Unfortunately even this didn't extract correctly field "Application data" even it should. This works correctly on regex101.com.

I'm testing this on Splunk 9.0.4.1.

isoutamo_0-1685102877611.png

 

0 Karma

SplunkDash
Motivator

Hello @isoutamo,

I know this is the best we can do for now. Thank you so much, truly appreciate your support.

0 Karma

SplunkDash
Motivator

@isoutamo ,

Thank you so much for your quick response, truly appreciate it. Let me try it and get back to you. Thank you again.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Edge Processor | New Resiliency Improvements & Support for Additional Data Sources

We are excited to announce several exciting updates for Edge Processor aimed at hardening overall product ...

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...