Json field parsing

[martinnepolean]

Hi,

We are getting the aws macie events as _json souretype, due to multiple loops there is a problem in fields extraction. I have give the screenshots below, red oval should be the field name and green oval should be valued.

for example the field name is detail.summary events.createtags.isp amazon and value is 436, but we need to field name till ISP and value to be amazon.

Please let me know how to get it gone through props and transforms

[to4kawa]

| makeresults 
| eval _raw="{\"version\":\"0\",\"id\":\"2561455-c673-0hy6-673b-447895415\",\"detail-type\":\"Macie Alert\",\"source\":\"aws.macie\",\"account\":\"123456789\",\"time\":\"2020-01-07T10:46:36Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\"],\"detail\":{\"notification-type\":\"ALERT_UPDATED\",\"tags\":[\"Suspicious Access\",\"Basic Alert\"],\"name\":\"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS\",\"severity\":\"LOW\",\"url\":\"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb\",\"alert-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"risk-score\":3,\"updated-at\":\"2020-01-07T10:46:36.136911\",\"created-at\":\"2020-01-07T00:46:35.139000+00:00\",\"actor\":\"321404829113:anonymous_principal\",\"summary\":{\"Description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"IP\":{\"216.20.176.6\":1,\"216.20.176.5\":1,\"216.20.176.4\":2,\"216.20.176.2\":3},\"Time Range\":[{\"count\":3,\"start\":\"2020-01-07T00:29:44Z\",\"end\":\"2020-01-07T00:29:47Z\"},{\"count\":1,\"start\":\"2020-01-07T10:06:11Z\",\"end\":\"2020-01-07T10:06:11Z\"},{\"count\":1,\"start\":\"2020-01-07T07:51:59Z\",\"end\":\"2020-01-07T07:51:59Z\"},{\"count\":1,\"start\":\"2020-01-07T10:19:18Z\",\"end\":\"2020-01-07T10:19:18Z\"},{\"count\":1,\"start\":\"2020-01-07T10:24:37Z\",\"end\":\"2020-01-07T10:24:37Z\"}],\"Record Count\":5,\"Location\":{\"us-east-1\":7},\"Event Count\":7,\"Events\":{\"GetObject\":{\"count\":5,\"ISP\":{\"Company\":5},\"Error Code\":{\"AccessDenied\":5}},\"ListObjects\":{\"count\":2,\"ISP\":{\"Company\":2},\"Error Code\":{\"AccessDenied\":2}}},\"recipientAccountId\":{\"321404829113\":7}},\"trigger\":{\"rule-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\",\"alert-type\":\"basic\",\"created-at\":\"2019-12-19 09:32:37.931000+00:00\",\"description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"risk\":3}}}"
| eval ErrorCode=spath(_raw,"detail.summary.Events.GetObject.Error Code")
| eval IP=spath(_raw,"detail.summary.IP")
| rex field=ErrorCode "(?<ErrorCode>\w+)"
| rex field=IP max_match=20 "(?<IP>(?<=\")[\w.]+)"
| fields - _*

Hi, @martinnepolean
The fields for the first question are missing, is this OK?

[martinnepolean]

I am looking to get all fields not only errorcode and IP. any help?

[kamlesh_vaghela]

@martinnepolean

Can you please share your expected output from the sample you have shared?

[martinnepolean]

Hi Kamlesh,

One of the example to show how the field extraction happens, currently we are getting like below one

Detail.Summary.Events.GetObject.Error Code.AccessDenied=2

But we like to extract it as
Detail.Summary.Events.GetObject.Error Code = AccessDenied

and Below is the list of fields I am looking for from the above event shared.

version=0
id=2561455-c673-0hy6-673b-447895415
detail-type=Maciealert
source=aws.macie
account=123456789
time=2020-01-07T10:46:36Z
region=us-east-1
resources=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
detail.notification-type=ALERT_UPDATED
tags=Suspicious Access,Basic Alert
name=Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS"
severity=LOW
url=https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%...
alert-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb
risk-score=3
updated-at=2020-01-07T10:46:36.136911
created-at=2020-01-07T00:46:35.139000+00:00
actor=321404829113:anonymous_principal
summary.Description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
IP=216.20.176.6,216.20.176.5,216.20.176.4,216.20.176.2
TimeRange="start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z","start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z","start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z","start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z","start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"
Location="us-east-1
Event Count=7
Events.GetObject.count=5
Events.GetObject.ISP=Company
Events.GetObject.Error Code=AccessDenied
Events.ListObjects.count=2
Events.ListObjects.ISP=Company
Events.ListObjects.Error Code=AccessDenied
recipientAccountId=321404829113
trigger.rule-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
trigger.alert-type=basic
trigger.created-at=2019-12-19 09:32:37.931000+00:00
trigger.description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
trigger.risk=3

[to4kawa]

spath, props.conf 's JSON extract is like this.
It is inevitable.

[to4kawa]

your sample doesn't have detail.summary events.createtags.isp
you say all fields. which?
Apparently the fields have been extracted.

[kamlesh_vaghela]

@martinnepolean
Can you please share sample _raw events and expected output?

[martinnepolean]

{"version":"0","id":"2561455-c673-0hy6-673b-447895415","detail-type":"Macie Alert","source":"aws.macie","account":"123456789","time":"2020-01-07T10:46:36Z","region":"us-east-1","resources":["arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"],"detail":{"notification-type":"ALERT_UPDATED","tags":["Suspicious Access","Basic Alert"],"name":"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS","severity":"LOW","url":"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%... Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","IP":{"216.20.176.6":1,"216.20.176.5":1,"216.20.176.4":2,"216.20.176.2":3},"Time Range":[{"count":3,"start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z"},{"count":1,"start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z"},{"count":1,"start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z"},{"count":1,"start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z"},{"count":1,"start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"}],"Record Count":5,"Location":{"us-east-1":7},"Event Count":7,"Events":{"GetObject":{"count":5,"ISP":{"Company":5},"Error Code":{"AccessDenied":5}},"ListObjects":{"count":2,"ISP":{"Company":2},"Error Code":{"AccessDenied":2}}},"recipientAccountId":{"321404829113":7}},"trigger":{"rule-arn":"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7","alert-type":"basic","created-at":"2019-12-19 09:32:37.931000+00:00","description":"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","risk":3}}}

I like to have error code as an field and AccessDenied as value which is not showing up now. and like to tag all IP addresses under IP field name.