Get dynamic json property names

tmortiboy — Fri, 22 Jul 2016 06:21:34 GMT

I have some structured json logs that indicate some validation errors, and depending on the error, a different property is set with the value that caused the error. eg:

{ "validationError": { "property" : { "title" : "" }, "errorType" : "Empty" } }
{ "validationError": { "property" : { "category" : [] }, "errorType" : "Empty" } }
{ "validationError": { "property" : { "category" : [ { "categoryId" : 12345 } ] }, "errorType" : "Invalid" } }
{ "validationError": { "property" : { "thing" : { "thingId" : 9999 } }, "errorType" : "Disallowed" } }

Re: Get dynamic json property names

javiergn — Mon, 25 Jul 2016 10:17:24 GMT

So assuming your field is called yourField, I came up with the following (probably overcomplicated) query:

| rex max_match=0 "(?m)(?<yourField>[^\n]+)"
| fields - _raw
| mvexpand yourField
| spath input=yourField path=validationError.property output=propertyRaw
| spath input=yourField path=validationError.errorType output=type
| fields - yourField
| spath input=propertyRaw
| rex field=propertyRaw max_match=1 "(?msi)\"(?<property>[^\"]+)\"\s?:"
| fields - propertyRaw
| eval value = ""
| eval temp = 'category{}.categoryId'
| foreach * [ eval temp = if (match("<<FIELD>>", property), toString('<<FIELD>>'), "") | eval value = value . temp]
| fields property, value, type
| foreach * [ eval <<FIELD>> = if(match(<<FIELD>>, "Null"), null(), <<FIELD>>) ]

Example:

| stats count | fields - count
| eval _raw = "
{ \"validationError\": { \"property\" : { \"title\" : \"\" }, \"errorType\" : \"Empty\" } }
{ \"validationError\": { \"property\" : { \"category\" : [] }, \"errorType\" : \"Empty\" } }
{ \"validationError\": { \"property\" : { \"category\" : [ { \"categoryId\" : 12345 } ] }, \"errorType\" : \"Invalid\" } }
{ \"validationError\": { \"property\" : { \"thing\" : { \"thingId\" : 9999 } }, \"errorType\" : \"Disallowed\" } }
"
| rex max_match=0 "(?m)(?<yourField>[^\n]+)"
| fields - _raw
| mvexpand yourField
| spath input=yourField path=validationError.property output=propertyRaw
| spath input=yourField path=validationError.errorType output=type
| fields - yourField
| spath input=propertyRaw
| rex field=propertyRaw max_match=1 "(?msi)\"(?<property>[^\"]+)\"\s?:"
| fields - propertyRaw
| eval value = ""
| eval temp = 'category{}.categoryId'
| foreach * [ eval temp = if (match("<<FIELD>>", property), toString('<<FIELD>>'), "") | eval value = value . temp]
| fields property, value, type
| foreach * [ eval <<FIELD>> = if(match(<<FIELD>>, "Null"), null(), <<FIELD>>) ]

Output, see picture below:

topic Get dynamic json property names in Getting Data In

Get dynamic json property names

Re: Get dynamic json property names