Getting Data In

How to do index time extraction of multiple fields inside another field which is in json format?

sivaranjiniG
Path Finder

I have a field message which have values has json format need to extract all the values in the json.

 

{ [-]
   guessedService: ejj
   logGroup: /aws/ejj/cluster
   logStream: kube-apt-15444d2f8c4b216a9cb69ac
   message:{"kind":"Event","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/jej/endpoints/eji.com-aws-eji","verb":"update","user":{"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]},"sourceIPs":["10.0.0.0"],"userAgent":"eji-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"endpoints","namespace":"edd","name":"dds.com-aws-edds","uid":"44ad8-899f-fbc1f4befb2f","apiVersion":"v1","resourceVersion":"8852157"},"responseStatus":{"metadata":{},"code":200}}

 

here from message field need to extract kind, stage, requestURI... and these fields inside json are dynamic(it can be more in other event). need help in extracting these fields in index time using props and transforms

 

Thanks

Labels (2)
Tags (1)
0 Karma
1 Solution

sivaranjiniG
Path Finder

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-un... 

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116 

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

 

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

View solution in original post

0 Karma

sivaranjiniG
Path Finder

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-un... 

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116 

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

 

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

0 Karma

FrankVl
Ultra Champion

That isn't index-time, but ok, a REPORT based solution might still be preferred over spath in the query. Except for the fact that json is way to complex to parse with such a simple regex. It doesn't support nesting, it doesn't support arrays: https://regex101.com/r/PJQJp7/1

This would work a bit better already: "([^"]+)":\s*("[^"]+|\[[^]]*\]|\d+)

https://regex101.com/r/u8hgTB/1

But that also doesn't extract json the way Splunk's auto extraction would. My solution does take content of nested structures, but without retaining that structure, it just captures all the 'leaves' in the json tree.

FrankVl
Ultra Champion

Any specific reason you need this as index-time extractions?

Any chance you could manipulate the overall format in which your receive the message? If the rest of the log format would have been valid json (which it almost is), you could have simply used the INDEXED_EXTRACTIONS = json setting in props.conf.

0 Karma

sivaranjiniG
Path Finder

we used spath in the query but its making the query heavy. thats the reason why trying to do it in the index time. 

All the available solution helps in extraction if whole event i json. but couldnt find anything to extract a fields from another field.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...