Getting Data In

How to do index time extraction of multiple fields inside another field which is in json format?

sivaranjiniG
Path Finder

I have a field message which have values has json format need to extract all the values in the json.

 

{ [-]
   guessedService: ejj
   logGroup: /aws/ejj/cluster
   logStream: kube-apt-15444d2f8c4b216a9cb69ac
   message:{"kind":"Event","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/jej/endpoints/eji.com-aws-eji","verb":"update","user":{"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]},"sourceIPs":["10.0.0.0"],"userAgent":"eji-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"endpoints","namespace":"edd","name":"dds.com-aws-edds","uid":"44ad8-899f-fbc1f4befb2f","apiVersion":"v1","resourceVersion":"8852157"},"responseStatus":{"metadata":{},"code":200}}

 

here from message field need to extract kind, stage, requestURI... and these fields inside json are dynamic(it can be more in other event). need help in extracting these fields in index time using props and transforms

 

Thanks

Labels (2)
Tags (1)
0 Karma
1 Solution

sivaranjiniG
Path Finder

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-un... 

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116 

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

 

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

View solution in original post

0 Karma

sivaranjiniG
Path Finder

Here is the solution..

Referred these,

https://community.splunk.com/t5/Getting-Data-In/Extract-JSON-data-within-the-logs-JSON-mixed-with-un... 

https://community.splunk.com/t5/Splunk-Search/Problem-with-using-SOURCE-KEY/m-p/40116 

Props.conf

[json_no_timestamp]
REPORT-json = report-json,report-json-kv
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

 

transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)

# Manually extract JSON key-value
[report-json-kv]
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
SOURCE_KEY = json2

0 Karma

FrankVl
Ultra Champion

That isn't index-time, but ok, a REPORT based solution might still be preferred over spath in the query. Except for the fact that json is way to complex to parse with such a simple regex. It doesn't support nesting, it doesn't support arrays: https://regex101.com/r/PJQJp7/1

This would work a bit better already: "([^"]+)":\s*("[^"]+|\[[^]]*\]|\d+)

https://regex101.com/r/u8hgTB/1

But that also doesn't extract json the way Splunk's auto extraction would. My solution does take content of nested structures, but without retaining that structure, it just captures all the 'leaves' in the json tree.

FrankVl
Ultra Champion

Any specific reason you need this as index-time extractions?

Any chance you could manipulate the overall format in which your receive the message? If the rest of the log format would have been valid json (which it almost is), you could have simply used the INDEXED_EXTRACTIONS = json setting in props.conf.

0 Karma

sivaranjiniG
Path Finder

we used spath in the query but its making the query heavy. thats the reason why trying to do it in the index time. 

All the available solution helps in extraction if whole event i json. but couldnt find anything to extract a fields from another field.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...