Extract fields from RFC5424 syslog with nested json field

[splunkreal]

Hello, I put this regex on SHC inline extraction : 

"<(?<pri>\d+)>1\s(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?[+-]\d{2}:\d{2})\s(?<hostname>[^\s]+)\s(?<appname>[^\s]+)\s(?<procid>[^\s]+)\s(?<msgid>[^\s]+)\s(?<structured_data>\S+)\s(?<json_msg>\{.*\})"

however json_msg needs | spath input=json_msg

Is it possible to auto extract fields contained in json_msg to avoid adding | spath input=json_msg at search time?

Thanks. 

[PickleRick]

Not at this time. Splunk can auto-extract values only if the whole _raw message consists of the structured data blob. There is an open idea on ideas.splunk.com - https://ideas.splunk.com/ideas/EID-I-208

It is marked as future prospect but of course voting on this issue might provide some additional push.

The alternative would be to cut the remainder of the event so that only the json part is left but this way you're losing some data.

[tej57]

Hello @splunkreal,

If you're using rex to extract the fields at search time, there's no way that Splunk will auto extract the fields that are part of your json_msg field. 

However, you can write the regex and have the fields extracted at search time using the field extraction from Settings -> Fields -> Field Extraction and define it under the sourcetype. So that everytime you run index based search, the fields json_msg will be extracted automatically and then you can use | spath json_msg to extract the subsequent fields. 

Alternatively, from the source if you are able to convert the whole string into JSON format, the nested json fields will be extracted automatically.

Regards,
Tejas.

---

If the above solution helps, an upvote is appreciated..!!