Getting Data In

Make extractions in props.conf from search query

to4kawa
Ultra Champion
| makeresults 
| eval _raw="Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"global\", \"origin\": \"dynstats\", \"values\": { } }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"imuxsock\", \"origin\": \"imuxsock\", \"submitted\": 0, \"ratelimit.discarded\": 0, \"ratelimit.numratelimiters\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 0\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 1\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }" 
| makemv delim="
 " _raw 
| stats count by _raw 
| rex "(?<json>{.*)" 
| spath input=json

This query works fine.
If I want to extract by props.conf, what's setting?

TIME_FORMAT = %B %d %T
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

I created it above, but I don't know the other settings.
If possible, please do not use SEDCMD and use it.

FIELD_HEADER_REGEX = ^.*?(?={)

Is this it?
cf. Extract fields from files with structured data

0 Karma
1 Solution

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

View solution in original post

0 Karma

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

0 Karma

to4kawa
Ultra Champion

props.conf

REPORT-json = json, json_field

transforms.conf

[json]
CLEAN_KEYS = 0
REGEX = (?P<jsontext>{.*)

[json_field]
CLEAN_KEYS = 0
REGEX = \"(\S+)\": \"?(\w+)\"?
FORMAT = $1::$2

\w+ of json_field needs to be changed depending on the situation.
This JSON is easy because it is not nested.

What if these are nested?

0 Karma

to4kawa
Ultra Champion

regex101

{ 
    "rsyslogd-pstats":[ 
        { 
            "name":"action 0",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"action 1",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"global",
            "origin":"dynstats",
            "values":{ 

            }
        },
        { 
            "name":"imuxsock",
            "origin":"imuxsock",
            "submitted":0,
            "ratelimit.discarded":0,
            "ratelimit.numratelimiters":0
        }
    ]
}

transforms.conf

SOURCEKEY = foo
REGEX = \"(\S+)\":\"?([\w.]+)\"?
MV_ADD = true

It seems that extraction is possible somehow.

0 Karma

adonio
Ultra Champion

it think the challenge with INDEXED_EXTRACTIONS there is that you have the timestamp first, so the file / event is not JSON per-se
you can use indexed extractions with transfoms.conf

0 Karma

to4kawa
Ultra Champion

Hi, @adonio ,Thank you for your reply .

I checked, FIELD_HEADER_REGEX looks different.

Which method is appropriate in this case?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...