topic Re: Indexed Extraction - all non-json events in Getting Data In

How to extract all log events excluding JSON messages?

nareshinsvu — Fri, 21 Aug 2020 21:58:20 GMT

Hi,

I want to extract all the log events (normal lines) except JSON messages. There should be an easy way for this. Any hints, please?

My log file is a mix something like below

----------

normal line

json events {

{json messages}

}

normal line

etc

Thanks,

Naresh

Re: Indexed Extraction - all non-json events

richgalloway — Fri, 21 Aug 2020 15:01:05 GMT

If you can produce a regular expression that defines a JSON event then you can use a transform to filter them out.

Put this in a tranforms.conf file

[indexdata] REGEX = . DEST_KEY = queue FORMAT = indexQueue [filterjson] REGEX = <your regex that detects JSON messages> DEST_KEY = queue FORMAT = nullQueue

Then add this to the corresponding props.conf file:

[mysourcetype] TRANSFORMS-nojson = indexdata, filterjson

Re: Indexed Extraction - all non-json events

nareshinsvu — Mon, 24 Aug 2020 03:25:25 GMT

Hi @richgalloway

I am struggling with regex actually. My regex is only capturing partial json message (until the first "}")

I am trying to search all lines between "line starting with {" and "line starting with }". But ^ is not picking my search

So, I am stuck with this regex currently -- \{[\s\S]*?\}

{

{},

Re: Indexed Extraction - all non-json events

to4kawa — Mon, 24 Aug 2020 08:36:34 GMT

@nareshinsvu

Regular expressions require a fairly strict definition. You haven't presented anything here.