topic Re: Field Extraction for Unstructured Quotation/Pair delimiter in Splunk Search

Field Extraction for Unstructured Quotation/Pair delimiter

SplunkDash — Wed, 22 Sep 2021 03:53:59 GMT

Hello,

How, I would write the regex for the following events (3 sample events provided below). It has "," pair delimiter, but " (quotation) are missing for one value (cit, shown in Bold) for some events. Any help will be highly appreciated, thank you.

"time_stamp":"2021-08-21 16:27:06 CDT","app_name":"CT-SATCUP","user_type":"TFilter","file_source_cd":"1","TFilterType":"0","ip_addr":"2300:1700:5c08:1030:6d93:7462:b15d:185c","session_id":"k/NJGhc8dU3OtYoRsrJ+pQzDdYE=","cit":"23235672174,"request_id":"32as3eee0a-0a31-6214a4e28-7e7fc700-6d792b5b203e","user_id":"cit1ddf82-bf36-40ca-84ae-7964b5680564","return_cd":"00","event_type":"SATCUP"

"time_stamp":"2021-08-21 16:27:05 CDT","app_name":"CT-SATCUP","user_type":"TFilter","file_source_cd":"1","TFilterType":"0","ip_addr":"2400:1700:5c08:1030:6d93:9462:b15d:185c","session_id":"k/NJGhc8dU3OtYoRsrJ+pQzDdYE=","cit":"232356756174","request_id":"31as3eee0a-0a31-6214a4e28-7e7fc700-6d792b5b203e","user_id":"cit1ddf82-bf36-40ca-84ae-7964b5680564","return_cd":"01","event_type":"SATCUP"

"time_stamp":"2021-08-21 16:27:08 CDT","app_name":"CT-SATCUP","user_type":"TFilter","file_source_cd":"4","TFilterType":"0","ip_addr":"2100:1700:5c08:1030:6r93:7462:b15d:185c","session_id":"k/NJGhc8dU3OtYoRsrJ+pQzDdYE=","cit":"232356756174,"request_id":"31bs3eee0a-0a31-6214a4e28-7e7fc700-6d792b5b203e","user_id":"cit1ddf82-bf36-40ca-84ae-7964b5680564","return_cd":"01","event_type":"SATCUP"

Re: Field Extraction for Unstructured Quotation/Pair delimiter

PickleRick — Wed, 22 Sep 2021 10:02:38 GMT

You can simply look for a field name in quotes and value not containing comma or quotation mark in possible quotes. Something like that

"(?<field_name>[^"]+)":"?(<?field_value>[^",]+)"?

EDIT: Watch out, there is indeed a typo here.

Re: Field Extraction for Unstructured Quotation/Pair delimiter

scelikok — Wed, 22 Sep 2021 07:08:30 GMT

@PickleRick,

There is a typo on second capture group;

"(?<field_name>[^"]+)":"?(?<field_value>[^",]+)"?

Re: Field Extraction for Unstructured Quotation/Pair delimiter

PickleRick — Wed, 22 Sep 2021 10:02:03 GMT

You're right. I was typing it on my tablet 🙂

Nice catch!

Re: Field Extraction for Unstructured Quotation/Pair delimiter

SplunkDash — Wed, 22 Sep 2021 13:16:38 GMT

Hello,

Thank you so much for the Regex codes you provided. Two points with this code

1. Field Names and Associated Field Values are coming under different groups

2. All Field Names are coming as "field_ name"

is there anyway we can pick the field_name from the events (i.e. time_stamp, app_name and so on) and put/keep associated field name and its field value together

Thank you again, appreciate your support in these efforts.

Re: Field Extraction for Unstructured Quotation/Pair delimiter

PickleRick — Wed, 22 Sep 2021 13:46:16 GMT

If you want dynamic field names, you might try something similar to this solution:

https://community.splunk.com/t5/Splunk-Search/How-to-extract-values/m-p/461411

Re: Field Extraction for Unstructured Quotation/Pair delimiter

SplunkDash — Wed, 22 Sep 2021 14:09:36 GMT

Thank you again, appreciated. But, just a little confused how I would incorporate this dynamic field names and associated field values selection code under this field extraction option/feature (see below the screenshot). Your support will be appreciated.....Thank you so much!!!

malekmo_0-1632319718422.png

Re: Field Extraction for Unstructured Quotation/Pair delimiter

PickleRick — Wed, 22 Sep 2021 19:18:23 GMT

Ahhh, that's kinda different. The solution I pointed you to was about parsing explicitly in your search. If you want to define such extraction as extracted fields - to be honest, I haven't done it using GUI but I suppose you have to define it using transforms (settings -> fields -> field transformations) with regex containing

"(?<_KEY_1>[^"]+)":"?(<?_VAL_1>[^",]+)"

And no Format

And then define field extraction using this transformation (change type from "inline" to "uses transformation")