topic Re: EVAL with REX with SPATH (props / transforms) in Getting Data In

EVAL with REX with SPATH (props / transforms)

willadams — Mon, 14 Sep 2020 08:22:36 GMT

I have a JSON file with an embedded JSON field that I am trying to extract. I have been doing some searching and have finally come up with an SPL search that will extract the information into my relevant key pairs. The SPL is basically

index=foo sourcetype=foosource | eval log_message=replace(log_message, "\\\"\"","") | rex "log_message\\\\\":\\\\\"(?<log_message>[^\\\]+)" | spath input=log_message

I have also read that I shouldn't need the spath however if I remove this from my SPL then it doesn't extract as required.

I would like to put this into transforms but unsure how to apply the "spath". Thoughts around my props/transforms so far is:

props.conf

REPORT-logmessage = log_message

transforms.conf

[log_message] INGEST_EVAL = eval_log_message=replace(log_message=replace(log_message, "\\\"\"","") REX = "log_message\\\\\":\\\\\"(?<log_message>[^\\\]+)"

Re: EVAL with REX with SPATH (props / transforms)

thambisetty — Mon, 14 Sep 2020 12:00:11 GMT

can you share sample event?

Re: EVAL with REX with SPATH (props / transforms)

willadams — Mon, 14 Sep 2020 12:48:40 GMT

{"log_message";"{\"version\":\"1.00000\",\"account_id\":\"12345678\",\"region\":\"myarea\",\" and so on until this blocks ends with }}". The remainder of the log after this is parsed appropriately "log_group":"/logs","log_location":"mylogs","account_information":"38430495"} for example.

Re: EVAL with REX with SPATH (props / transforms)

thambisetty — Mon, 14 Sep 2020 14:08:24 GMT

@willadams ,

I can't give you exact answer for your question, but I can direct you the approach to be taken.

let me explain what I understand from your question:

you have new json which is log_message inside json event. you want to extract fields of log_message automatically.

I have taken sample event as below : cars is json inside json event

{ "name":"John", "age":30, "cars": { "carname":"Ford", "models":"Fiesta" } }

first, I have extracted to cars value "{ "carname":"Ford", "models":"Fiesta" }" to cars using props.conf below. [jsontest] is my sourcetype of event.

created a report extractcars to be used in transforms.conf

props.conf

[jsontest] EXTRACT-cars = \"cars\":\s+\{\s(?<cars>[^\}]+) REPORT-extractcars = extractcars

and then I have extracted fields carname and models using transforms.conf

SOURCE_KEY is key to define for which field you would to apply REGEX

transforms.conf

[extractcars] REGEX = \"(?<_KEY_1>[^\"]+)\"\:\"(?<_VAL_1>[^\"]+) SOURCE_KEY = cars

Re: EVAL with REX with SPATH (props / transforms)

willadams — Mon, 14 Sep 2020 15:05:57 GMT

I will give a shot but I have to do the replace first before the rex.