Getting Data In

Make extractions in props.conf from search query

to4kawa
Ultra Champion
| makeresults 
| eval _raw="Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"global\", \"origin\": \"dynstats\", \"values\": { } }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"imuxsock\", \"origin\": \"imuxsock\", \"submitted\": 0, \"ratelimit.discarded\": 0, \"ratelimit.numratelimiters\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 0\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }
 Nov 14 03:23:42 hostname rsyslogd-pstats:{ \"name\": \"action 1\", \"origin\": \"core.action\", \"processed\": 50996, \"failed\": 0, \"suspended\": 0, \"suspended.duration\": 0, \"resumed\": 0 }" 
| makemv delim="
 " _raw 
| stats count by _raw 
| rex "(?<json>{.*)" 
| spath input=json

This query works fine.
If I want to extract by props.conf, what's setting?

TIME_FORMAT = %B %d %T
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

I created it above, but I don't know the other settings.
If possible, please do not use SEDCMD and use it.

FIELD_HEADER_REGEX = ^.*?(?={)

Is this it?
cf. Extract fields from files with structured data

0 Karma
1 Solution

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

View solution in original post

0 Karma

to4kawa
Ultra Champion

props.conf

[json_sed]
TIME_FORMAT = %B %d %T
SEDCMD = s/.*?({.*)/\1/g
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true

This setting is OK.
I was worried that time information would be lost by using SEDCMD ,but there was no problem.

KV_MODE and INDEXED_EXTRACTIONS needs valid format.
If we have invalid json , we should make them to valid format.

0 Karma

to4kawa
Ultra Champion

props.conf

REPORT-json = json, json_field

transforms.conf

[json]
CLEAN_KEYS = 0
REGEX = (?P<jsontext>{.*)

[json_field]
CLEAN_KEYS = 0
REGEX = \"(\S+)\": \"?(\w+)\"?
FORMAT = $1::$2

\w+ of json_field needs to be changed depending on the situation.
This JSON is easy because it is not nested.

What if these are nested?

0 Karma

to4kawa
Ultra Champion

regex101

{ 
    "rsyslogd-pstats":[ 
        { 
            "name":"action 0",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"action 1",
            "origin":"core.action",
            "processed":50996,
            "failed":0,
            "suspended":0,
            "suspended.duration":0,
            "resumed":0
        },
        { 
            "name":"global",
            "origin":"dynstats",
            "values":{ 

            }
        },
        { 
            "name":"imuxsock",
            "origin":"imuxsock",
            "submitted":0,
            "ratelimit.discarded":0,
            "ratelimit.numratelimiters":0
        }
    ]
}

transforms.conf

SOURCEKEY = foo
REGEX = \"(\S+)\":\"?([\w.]+)\"?
MV_ADD = true

It seems that extraction is possible somehow.

0 Karma

adonio
Ultra Champion

it think the challenge with INDEXED_EXTRACTIONS there is that you have the timestamp first, so the file / event is not JSON per-se
you can use indexed extractions with transfoms.conf

0 Karma

to4kawa
Ultra Champion

Hi, @adonio ,Thank you for your reply .

I checked, FIELD_HEADER_REGEX looks different.

Which method is appropriate in this case?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...