topic Re: Create key value pairs from existing fields in Splunk Search

Create key value pairs from existing fields

ben_leung — Sat, 01 Nov 2014 01:15:24 GMT

In _raw:

I want to manipulate/rearrange and get an output that is like this.

key1=value1
key2=value2
key3=value3
...

Re: Create key value pairs from existing fields

Raghav2384 — Mon, 28 Sep 2020 18:03:53 GMT

May be this...

Re: Create key value pairs from existing fields

jkat54 — Sat, 01 Nov 2014 06:03:53 GMT

Hi, this should get you in the right direction:

...|rex field=_raw "string1=(?<field1>\w+\d+)\|(?<field3>\w+\d+)\|(?<field5>\w+\d+)\|(?<field7>\w+\d+)\|(?<field9>\w+\d+)\|(?<field11>\w+\d+)\sstring2=(?<field2>\w+\d+)\|(?<field4>\w+\d+)\|(?<field6>\w+\d+)\|(?<field8>\w+\d+)\|(?<field10>\w+\d+)\|(?<field12>\w+\d+)$" | eval _raw=field1."=".field2.",".field3."=".field4.",".field5."=".field6.",".field7."=".field8.",".field9."=".field10.",".field11."=".field12

Re: Create key value pairs from existing fields

ben_leung — Mon, 03 Nov 2014 03:03:21 GMT

base search| makemv delim="|" System1|makemv delim="|" System2|eval fields = mvzip(System1,System2)|mvexpand fields

The new "fields" field has what I need but only gets the first fields="key1=vlaue1". The other fields and values are not grabbed since the mvzip argument in the eval command only grabs the first.

Not sure what is the purpose for the rex command and everything after that.

Re: Create key value pairs from existing fields

ben_leung — Mon, 03 Nov 2014 03:08:46 GMT

The challenge is that for string1 and string 2, there are random number of values, although they will have the same number of them.

There are 3 in the first event and then 2 in the next.

2014-10-23 22:25:41 string1=key1|key2|key3  string2=value1|value2|value3
2014-10-23 22:26:00 string1=key4|key5  string2=value4|value5

Re: Create key value pairs from existing fields

jkat54 — Fri, 07 Nov 2014 03:43:03 GMT

Dude... what is up with this data!

This is probably easier another way, but this example supports up to 10 key values and rewrites the timestamp as well, note i separate the date from the other strings with a colon in the eval, you might want to change it to a space. Finally we remove the extra ,= if they exist using sed:

...|rex field=_raw "(^.*(?=string1=))string1=(?<f1>\w+)?\|?(?<f2>\w+)?\|?(?<f3>\w+)?\|?(?<f4>\w+)?\|?(?<f5>\w+)?\|?(?<f6>\w+)?\|?(?<f7>\w+)?\|?(?<f8>\w+)?\|?(?<f9>\w+)?\|?(?<f10>\w+)?\|?\s+string2=(?<f11>\w+)?\|?(?<f12>\w+)?\|?(?<f13>\w+)?\|?(?<f14>\w+)?\|?(?<f15>\w+)?\|?(?<f16>\w+)?\|?(?<f17>\w+)?\|?(?<f18>\w+)?\|?(?<f19>\w+)?\|?(?<f20>\w+)?\|?$" | eval _raw=f1.":".f2.=".f12.",".f3."=".f13.",".f4."=".f14.",".f5."=".f15.",".f6."=".f16.",".f7."=".f17.",".f8."=".f18.",".f9."=".f19.",".f10."=".f20.",".f11."=".f21 | rex mode=sed field=_raw "s/(\,=)+//g"

Works in theory!

Re: Create key value pairs from existing fields

ben_leung — Fri, 07 Nov 2014 05:00:07 GMT

The office politics with developers..

Re: Create key value pairs from existing fields

jkat54 — Fri, 07 Nov 2014 16:36:45 GMT

does the crazy expensive search above work for now? Are you ok with the commas between the key value pairs or did you absolutely need line breaks?

Re: Create key value pairs from existing fields

somesoni2 — Fri, 07 Nov 2014 17:31:16 GMT

If this is your _raw data

2014-10-23 22:25:41 string1=key1|key2|key3  string2=value1|value2|value3
 2014-10-23 22:26:00 string1=key4|key5  string2=value4|value5

then try this (runanywhere sample, till eval _time=now() is just for generating dummy data, this will be your base search)

| gentimes start=-1 | eval temp="2014-10-23 22:25:41 string1=key1|key2|key3  string2=value1|value2|value3#2014-10-23 22:26:00 string1=key4|key5  string2=value4|value5" | table temp | makemv temp delim="#" | mvexpand temp | rename temp as _raw| extract pairdelim=" ", kvdelim="=:" | eval _time=now()
| eval string1="timestamp|".string1 | eval string2=_time."|".string2 | makemv delim="|" string1 | makemv delim="|" string2 | eval temp=mvzip(string1,string2,"=") | nomv temp | rename temp as _raw | table _raw| extract pairdelim=" ", kvdelim="=:"