topic Re: Is it possible to parse an extracted field as json if the whole log line isn't json? in Getting Data In

Is it possible to parse an extracted field as json if the whole log line isn't json?

stevennoble — Tue, 22 Oct 2013 22:38:56 GMT

If I have a line of my logs that look something like

[2013-10-18 23:36:50.785476] {"message":"some message", "headers": {"a": 1, "b": 2}}

is there a way I can use splunk's regex extraction to separate the timestamp from the json, then use splunk's json extraction to to extract fields from the json?

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

_d_ — Tue, 22 Oct 2013 23:11:51 GMT

You can't use KV_MODE=json in props.conf because in its current state the event is not fully json. Additionally you can't extract the rest of the messages and then use the same setting on it (again, from props.conf). However, you can do it inline with spath.

Extract the whole json message in a field called, say, my_field, then use spath:

...| spath input=my_field

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

stevennoble — Tue, 22 Oct 2013 23:19:17 GMT

This is something that I'd use in a search? Or can this be applied to a log source?

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

_d_ — Tue, 22 Oct 2013 23:22:13 GMT

Inline in a search. You can follow the ..|spath input=my_field with any other search commands. More details and examples here: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

kamermans — Mon, 28 Sep 2020 17:04:38 GMT

You can remove the string leader using any the following methods:

Remove leader by pretending it's a line breaker
LINE_BREAKER=((:?^|\n).+?){ SHOULD_LINEMERGE=false
Removing the leader with SEDCMD:
SEDCMD-StripHeader=s/^[^{]+//
Removing the leader via a transform on _raw:
; in transforms.conf: [StripSyslog] REGEX = ^[^{]+(.*)$ FORMAT = $1 DEST_KEY = _raw
; in props.conf: TRANSFORMS-StripSyslog = StripSyslog

All these methods will work with KV_MODE=json, but note that currently they will not work with INDEXED_EXTRACTIONS=json.

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

wsnyder2 — Fri, 30 Oct 2015 14:51:50 GMT

This is a bummer ... We have jason as part of an event. Would love to have field extraction happen under the hood, e.g. in props.conf or transforms.conf.
Seems like there is a way based on postings in this thread. (?)

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

wsnyder2 — Fri, 30 Oct 2015 15:00:52 GMT

Can I do this and keep my header?

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

prees — Tue, 08 Dec 2015 21:29:39 GMT

Was any one able to do this? as originally described, ie. without using spath?

Re: Is it possible to parse an extracted field as json if the whole log line isn't json?

prees — Tue, 08 Dec 2015 21:39:16 GMT

Second what @wsnyder2 said. Is it possible to do this and still retain the info from the header?

EDIT: or is it possible to pre-transform them based on the header then apply this transform?