topic Re: How to do index time extraction of multiple fields inside another field which is in json format? in Getting Data In

How to do index time extraction of multiple fields inside another field which is in json format?

sivaranjiniG — Wed, 17 Nov 2021 08:56:41 GMT

I have a field message which have values has json format need to extract all the values in the json.

{ [-] guessedService: ejj logGroup: /aws/ejj/cluster logStream: kube-apt-15444d2f8c4b216a9cb69ac message:{"kind":"Event","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/jej/endpoints/eji.com-aws-eji","verb":"update","user":{"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]},"sourceIPs":["10.0.0.0"],"userAgent":"eji-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"endpoints","namespace":"edd","name":"dds.com-aws-edds","uid":"44ad8-899f-fbc1f4befb2f","apiVersion":"v1","resourceVersion":"8852157"},"responseStatus":{"metadata":{},"code":200}}

here from message field need to extract kind, stage, requestURI... and these fields inside json are dynamic(it can be more in other event). need help in extracting these fields in index time using props and transforms

Thanks

Re: How to do index time extraction of multiple fields inside another field which is in json format?

FrankVl — Wed, 17 Nov 2021 15:35:41 GMT

Any specific reason you need this as index-time extractions?

Any chance you could manipulate the overall format in which your receive the message? If the rest of the log format would have been valid json (which it almost is), you could have simply used the INDEXED_EXTRACTIONS = json setting in props.conf.

Re: How to do index time extraction of multiple fields inside another field which is in json format?

sivaranjiniG — Sun, 21 Nov 2021 12:00:27 GMT

we used spath in the query but its making the query heavy. thats the reason why trying to do it in the index time.

All the available solution helps in extraction if whole event i json. but couldnt find anything to extract a fields from another field.

Re: How to do index time extraction of multiple fields inside another field which is in json format?

sivaranjiniG — Sun, 21 Nov 2021 15:49:03 GMT

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-unstructured/td-p/195292

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

Re: How to do index time extraction of multiple fields inside another field which is in json format?

FrankVl — Mon, 22 Nov 2021 08:09:28 GMT

That isn't index-time, but ok, a REPORT based solution might still be preferred over spath in the query. Except for the fact that json is way to complex to parse with such a simple regex. It doesn't support nesting, it doesn't support arrays: https://regex101.com/r/PJQJp7/1

This would work a bit better already: "([^"]+)":\s*("[^"]+|\[[^]]*\]|\d+)

https://regex101.com/r/u8hgTB/1

But that also doesn't extract json the way Splunk's auto extraction would. My solution does take content of nested structures, but without retaining that structure, it just captures all the 'leaves' in the json tree.