How to do index time extraction of multiple fields inside another field which is in json format?

[sivaranjiniG]

I have a field message which have values has json format need to extract all the values in the json.

{ [-]
   guessedService: ejj
   logGroup: /aws/ejj/cluster
   logStream: kube-apt-15444d2f8c4b216a9cb69ac
   message:{"kind":"Event","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/jej/endpoints/eji.com-aws-eji","verb":"update","user":{"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]},"sourceIPs":["10.0.0.0"],"userAgent":"eji-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"endpoints","namespace":"edd","name":"dds.com-aws-edds","uid":"44ad8-899f-fbc1f4befb2f","apiVersion":"v1","resourceVersion":"8852157"},"responseStatus":{"metadata":{},"code":200}}

here from message field need to extract kind, stage, requestURI... and these fields inside json are dynamic(it can be more in other event). need help in extracting these fields in index time using props and transforms

Thanks

[sivaranjiniG · Accepted solution]

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-un... 

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116 

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

[FrankVl]

That isn't index-time, but ok, a REPORT based solution might still be preferred over spath in the query. Except for the fact that json is way to complex to parse with such a simple regex. It doesn't support nesting, it doesn't support arrays: https://regex101.com/r/PJQJp7/1

This would work a bit better already: "([^"]+)":\s*("[^"]+|\[[^]]*\]|\d+)

https://regex101.com/r/u8hgTB/1

But that also doesn't extract json the way Splunk's auto extraction would. My solution does take content of nested structures, but without retaining that structure, it just captures all the 'leaves' in the json tree.

[FrankVl]

Any specific reason you need this as index-time extractions?

Any chance you could manipulate the overall format in which your receive the message? If the rest of the log format would have been valid json (which it almost is), you could have simply used the INDEXED_EXTRACTIONS = json setting in props.conf.

[sivaranjiniG]

we used spath in the query but its making the query heavy. thats the reason why trying to do it in the index time. 

All the available solution helps in extraction if whole event i json. but couldnt find anything to extract a fields from another field.