Extracting key-value pairs using regex at search time

jmartens — Thu, 10 Dec 2020 08:01:55 GMT

I am trying to extract multiple key value pairs from data like this:

So far I have tried the following:

index="test" | rex max_match=0 field=_raw "(?<k1>[^\|\s]+)\s+\|(?<k2>[^\|]+)\|(?<v>[^\|]+)[\|]?[\r\n]" | eval z=mvzip(mvzip(trim(k1), trim(k2), " "), trim(v), "~") | mvexpand z | rex field=z "(?<key>[^~]+)~(?<value>.*)" | eval {key} = value

It yields desired results for key-value pairs, but each individual key value pair is assigned to an individual copy of the event, thus generating a cross product of events and extracted keys.

I don't want that. I want all extractions to be tied to the unique event they are extracted from.

So I tried this:

index="test" | rex max_match=0 field=_raw "(?<k1>[^\|]+)\|(?<k2>[^\|]+)\|(?<v>[^\|]+)[\|]?[\r\n]+" | eval key=mvzip(trim(k1), trim(k2), " ") | eval {key}=v

Which extracts the key and values properly, but when trying to evaluate the key=value statement it merges all key values into the field name and assigns each individual value to it, so now I have separate values but no unique keys and my key looks like this:

Image Loading Image Photometric Interpretation Image Compression Image Image width [mm] Image Image height [mm] Image Bit-Depth Image Row Vector [mm] Image Column Vector [mm] Image Image Position [mm] Image PixelsizeX [mm] Image PixelsizeY [mm] Image Img Orientation Info Image Horizontal flip Image Vertical flip Image Brainlab Subsystem Image InstNumber/BLScanNum Image Series number Image Instance UID Image Study UID Image Series UID Image BitsAllocated Image PatientOrientation Image HeadFeetOrientation Image Modality Image SliceThickness [mm] Image AcquisitionID Image FrameOfRefUID Image ScanDate Image ScanTime Image Manufacturer Image Manufacturer model Image Institution name Image Station name Image Software Versions Image Comment

How can I get the best of both?

evaluated keys from search time extraction
value assigned to the proper key
all key-value pairs connected to the unique individual event

Re: Extracting key-value pairs using regex at search time

jmartens — Thu, 10 Dec 2020 09:10:15 GMT

Eventually I was able to solve it using redirection of the _raw event with a custom string and parsing that with extract, making sure to use custom delimters not present in my dataset like this:

index="test" | eval temp=_raw | rex max_match=0 field=_raw "(?<k1>[^\|]+)\|(?<k2>[^\|]+)\|(?<v>[^\|]+)[\|]?[\r\n]+" | eval _raw=mvjoin(mvzip(mvzip(trim(k1), trim(k2), " "), trim(v), "#"), "@") | extract pairdelim="@" kvdelim="#" clean_keys=f | eval _raw=temp

topic Re: Extracting key-value pairs using regex at search time in Splunk Search

Extracting key-value pairs using regex at search time

Re: Extracting key-value pairs using regex at search time