Field extraction using props and transforms

[sivaranjiniG]

Hello,

{ [-]
   guessedService: ejj
   logGroup: /aws/ejj/cluster
   logStream: kube-apt-15444d2f8c4b216a9cb69ac
   message:{"kind":"Event","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/jej/endpoints/eji.com-aws-eji","verb":"update","user":{"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]},"sourceIPs":["10.0.0.0"],"userAgent":"eji-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"endpoints","namespace":"edd","name":"dds.com-aws-edds","uid":"44ad8-899f-fbc1f4befb2f","apiVersion":"v1","resourceVersion":"8852157"},"responseStatus":{"metadata":{},"code":200}}

i already a below props and transforms to extract all the fields from message. 

Props.conf
[json_no_new]
REPORT-json = report-json,report-json-new
KV_MODE = none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ^{
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

Transforms.conf

[report-json]
SOURCE_KEY = message
REGEX = (?P<json2>{.+)
DEST_KEY = _raw

[report-json-new]
REGEX = \\*"([^"]+)\":[\s]*"*(\[.*?\]|\{.*?\}"*\}*|[^"]+|\d+),*
FORMAT = $1::$2
SOURCE_KEY = json2

Now from the result i have below field with json value

user = {"username":"system:serviceaccount:efs:efs-provisioner","uid":"ab5d27b4c-71a4f77323b0","groups":["system:serviceaccounts","system:serviceaccounts:eji","system:authenticated"]}

again with props and transform i want to extract values from user field.

Please some one let me know if thats possible 

Thanks

[VatsalJagani]

You should be getting all the fields being extracted just with INDEXED_EXTRACTION. As your data in proper JSON format, you don't even need those transforms.

You should see fields like: logGroup, message.kind, message.user.username, message.user.uid, etc.

Though alternatively, you can use search time extraction, which is what I would do: Using KV_MODE=json instead of INDEXED_EXTRACTION=json. 

Try this below configuration if you can on test system:

[json_no_new]
KV_MODE = json
LINE_BREAKER = }([\r\n]+)
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

You may need to change the SHOULD_LINEMERGE along with other configurations to make sure the data being extracted in the right events according to your _raw data.

You should see fields like with search-time extraction as well: logGroup, message.kind, message.user.username, message.user.uid, etc.

Fields are hierarchical with the use of .(dot).

[PickleRick]

As far as I remember, the automatic json extraction (contrary to the spath command) does not care about attributes hierarchy.