EVAL with REX with SPATH (props / transforms)

[willadams]

I have a JSON file with an embedded JSON field that I am trying to extract.  I have been doing some searching and have finally come up with an SPL search that will extract the information into my relevant key pairs.  The SPL is basically

index=foo sourcetype=foosource
| eval log_message=replace(log_message, "\\\"\"","")
| rex "log_message\\\\\":\\\\\"(?<log_message>[^\\\]+)"
| spath input=log_message

I have also read that I shouldn't need the spath however if I remove this from my SPL then it doesn't extract as required.

I would like to put this into transforms but unsure how to apply the "spath".  Thoughts around my props/transforms so far is:

props.conf

REPORT-logmessage = log_message

transforms.conf

[log_message]
INGEST_EVAL = eval_log_message=replace(log_message=replace(log_message, "\\\"\"","")
REX = "log_message\\\\\":\\\\\"(?<log_message>[^\\\]+)"

[willadams]

{"log_message";"{\"version\":\"1.00000\",\"account_id\":\"12345678\",\"region\":\"myarea\",\" and so on until this blocks ends with }}".  The remainder of the log after this is parsed appropriately "log_group":"/logs","log_location":"mylogs","account_information":"38430495"} for example.

[thambisetty]

@willadams ,

I can't give you exact answer for your question, but I can direct you the approach to be taken.

let me explain what I understand from your question:

you have new json which is log_message inside json event. you want to extract fields of log_message automatically.

I have taken sample event as below : cars is json inside json event

{
  "name":"John",
  "age":30,
  "cars":
    { "carname":"Ford", "models":"Fiesta" }
 }

first, I have extracted to cars value "{ "carname":"Ford", "models":"Fiesta" }" to cars  using props.conf below. [jsontest] is my sourcetype of event.

created a report extractcars to be used in transforms.conf

props.conf

[jsontest]
EXTRACT-cars = \"cars\":\s+\{\s(?<cars>[^\}]+)
REPORT-extractcars = extractcars

and then I have extracted fields carname and models using transforms.conf

SOURCE_KEY is key to define for which field you would to apply REGEX

transforms.conf

[extractcars]
REGEX = \"(?<_KEY_1>[^\"]+)\"\:\"(?<_VAL_1>[^\"]+)
SOURCE_KEY = cars

[willadams]

I will give a shot but I have to do the replace first before the rex. 

[thambisetty]

can you share sample event?