topic Re: Make extractions in props.conf from search query in Getting Data In

Make extractions in props.conf from search query

to4kawa — Sun, 09 Feb 2020 01:26:12 GMT

| makeresults 
| eval _raw="Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"global\", \"origin\": \"dynstats\", \"values\": { } }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"imuxsock\", \"origin\": \"imuxsock\", \"submitted\": 0, \"ratelimit.discarded\": 0, \"ratelimit.numratelimiters\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 0\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 1\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }" 
| makemv delim="
 " _raw 
| stats count by _raw 
| rex "(?<json>{.*)" 
| spath input=json

This query works fine.
If I want to extract by props.conf, what's setting?

TIME_FORMAT = %B %d %T
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

I created it above, but I don't know the other settings.
If possible, please do not use SEDCMD and use it.

FIELD_HEADER_REGEX = ^.*?(?={)

Is this it?
cf. Extract fields from files with structured data

Re: Make extractions in props.conf from search query

adonio — Sun, 09 Feb 2020 03:40:29 GMT

it think the challenge with INDEXED_EXTRACTIONS there is that you have the timestamp first, so the file / event is not JSON per-se
you can use indexed extractions with transfoms.conf

Re: Make extractions in props.conf from search query

to4kawa — Sun, 09 Feb 2020 10:09:26 GMT

Hi, @adonio ,Thank you for your reply .

I checked, FIELD_HEADER_REGEX looks different.

Which method is appropriate in this case?

Re: Make extractions in props.conf from search query

to4kawa — Sat, 15 Feb 2020 03:03:09 GMT

props.conf

REPORT-json = json, json_field

transforms.conf

[json]
CLEAN_KEYS = 0
REGEX = (?P<jsontext>{.*)

[json_field]
CLEAN_KEYS = 0
REGEX = \"(\S+)\": \"?(\w+)\"?
FORMAT = $1::$2

\w+ of json_field needs to be changed depending on the situation.
This JSON is easy because it is not nested.

What if these are nested?

Re: Make extractions in props.conf from search query

to4kawa — Sat, 22 Feb 2020 03:47:13 GMT

regex101

{ 
    "rsyslogd-pstats":[ 
        { 
            "name":"action 0",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"action 1",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"global",
            "origin":"dynstats",
            "values":{ 

            }
        },
        { 
            "name":"imuxsock",
            "origin":"imuxsock",
            "submitted":0,
            "ratelimit.discarded":0,
            "ratelimit.numratelimiters":0
        }
    ]
}

transforms.conf

SOURCEKEY = foo
REGEX = \"(\S+)\":\"?([\w.]+)\"?
MV_ADD = true

It seems that extraction is possible somehow.

Re: Make extractions in props.conf from search query

to4kawa — Sun, 01 Mar 2020 03:41:19 GMT

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.