Getting Data In

Make extractions in props.conf from search query

to4kawa
Ultra Champion
| makeresults 
| eval _raw="Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"global\", \"origin\": \"dynstats\", \"values\": { } }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"imuxsock\", \"origin\": \"imuxsock\", \"submitted\": 0, \"ratelimit.discarded\": 0, \"ratelimit.numratelimiters\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 0\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 1\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }" 
| makemv delim="
 " _raw 
| stats count by _raw 
| rex "(?<json>{.*)" 
| spath input=json

This query works fine.
If I want to extract by props.conf, what's setting?

TIME_FORMAT = %B %d %T
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

I created it above, but I don't know the other settings.
If possible, please do not use SEDCMD and use it.

FIELD_HEADER_REGEX = ^.*?(?={)

Is this it?
cf. Extract fields from files with structured data

0 Karma
1 Solution

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

View solution in original post

0 Karma

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

0 Karma

to4kawa
Ultra Champion

props.conf

REPORT-json = json, json_field

transforms.conf

[json]
CLEAN_KEYS = 0
REGEX = (?P<jsontext>{.*)

[json_field]
CLEAN_KEYS = 0
REGEX = \"(\S+)\": \"?(\w+)\"?
FORMAT = $1::$2

\w+ of json_field needs to be changed depending on the situation.
This JSON is easy because it is not nested.

What if these are nested?

0 Karma

to4kawa
Ultra Champion

regex101

{ 
    "rsyslogd-pstats":[ 
        { 
            "name":"action 0",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"action 1",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"global",
            "origin":"dynstats",
            "values":{ 

            }
        },
        { 
            "name":"imuxsock",
            "origin":"imuxsock",
            "submitted":0,
            "ratelimit.discarded":0,
            "ratelimit.numratelimiters":0
        }
    ]
}

transforms.conf

SOURCEKEY = foo
REGEX = \"(\S+)\":\"?([\w.]+)\"?
MV_ADD = true

It seems that extraction is possible somehow.

0 Karma

adonio
Ultra Champion

it think the challenge with INDEXED_EXTRACTIONS there is that you have the timestamp first, so the file / event is not JSON per-se
you can use indexed extractions with transfoms.conf

0 Karma

to4kawa
Ultra Champion

Hi, @adonio ,Thank you for your reply .

I checked, FIELD_HEADER_REGEX looks different.

Which method is appropriate in this case?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...