topic Re: Json field parsing in Getting Data In

Json field parsing

martinnepolean — Mon, 06 Jan 2020 13:04:40 GMT

Hi,

We are getting the aws macie events as _json souretype, due to multiple loops there is a problem in fields extraction. I have give the screenshots below, red oval should be the field name and green oval should be valued.

for example the field name is detail.summary events.createtags.isp amazon and value is 436, but we need to field name till ISP and value to be amazon.

Please let me know how to get it gone through props and transforms

Re: Json field parsing

kamlesh_vaghela — Tue, 07 Jan 2020 05:35:22 GMT

@martinnepolean
Can you please share sample _raw events and expected output?

Re: Json field parsing

martinnepolean — Wed, 30 Sep 2020 03:32:21 GMT

{"version":"0","id":"2561455-c673-0hy6-673b-447895415","detail-type":"Macie Alert","source":"aws.macie","account":"123456789","time":"2020-01-07T10:46:36Z","region":"us-east-1","resources":["arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"],"detail":{"notification-type":"ALERT_UPDATED","tags":["Suspicious Access","Basic Alert"],"name":"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS","severity":"LOW","url":"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb","alert-arn":"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","risk-score":3,"updated-at":"2020-01-07T10:46:36.136911","created-at":"2020-01-07T00:46:35.139000+00:00","actor":"321404829113:anonymous_principal","summary":{"Description":"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","IP":{"216.20.176.6":1,"216.20.176.5":1,"216.20.176.4":2,"216.20.176.2":3},"Time Range":[{"count":3,"start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z"},{"count":1,"start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z"},{"count":1,"start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z"},{"count":1,"start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z"},{"count":1,"start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"}],"Record Count":5,"Location":{"us-east-1":7},"Event Count":7,"Events":{"GetObject":{"count":5,"ISP":{"Company":5},"Error Code":{"AccessDenied":5}},"ListObjects":{"count":2,"ISP":{"Company":2},"Error Code":{"AccessDenied":2}}},"recipientAccountId":{"321404829113":7}},"trigger":{"rule-arn":"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7","alert-type":"basic","created-at":"2019-12-19 09:32:37.931000+00:00","description":"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","risk":3}}}

I like to have error code as an field and AccessDenied as value which is not showing up now. and like to tag all IP addresses under IP field name.

Re: Json field parsing

to4kawa — Tue, 07 Jan 2020 12:04:57 GMT

| makeresults 
| eval _raw="{\"version\":\"0\",\"id\":\"2561455-c673-0hy6-673b-447895415\",\"detail-type\":\"Macie Alert\",\"source\":\"aws.macie\",\"account\":\"123456789\",\"time\":\"2020-01-07T10:46:36Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\"],\"detail\":{\"notification-type\":\"ALERT_UPDATED\",\"tags\":[\"Suspicious Access\",\"Basic Alert\"],\"name\":\"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS\",\"severity\":\"LOW\",\"url\":\"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb\",\"alert-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"risk-score\":3,\"updated-at\":\"2020-01-07T10:46:36.136911\",\"created-at\":\"2020-01-07T00:46:35.139000+00:00\",\"actor\":\"321404829113:anonymous_principal\",\"summary\":{\"Description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"IP\":{\"216.20.176.6\":1,\"216.20.176.5\":1,\"216.20.176.4\":2,\"216.20.176.2\":3},\"Time Range\":[{\"count\":3,\"start\":\"2020-01-07T00:29:44Z\",\"end\":\"2020-01-07T00:29:47Z\"},{\"count\":1,\"start\":\"2020-01-07T10:06:11Z\",\"end\":\"2020-01-07T10:06:11Z\"},{\"count\":1,\"start\":\"2020-01-07T07:51:59Z\",\"end\":\"2020-01-07T07:51:59Z\"},{\"count\":1,\"start\":\"2020-01-07T10:19:18Z\",\"end\":\"2020-01-07T10:19:18Z\"},{\"count\":1,\"start\":\"2020-01-07T10:24:37Z\",\"end\":\"2020-01-07T10:24:37Z\"}],\"Record Count\":5,\"Location\":{\"us-east-1\":7},\"Event Count\":7,\"Events\":{\"GetObject\":{\"count\":5,\"ISP\":{\"Company\":5},\"Error Code\":{\"AccessDenied\":5}},\"ListObjects\":{\"count\":2,\"ISP\":{\"Company\":2},\"Error Code\":{\"AccessDenied\":2}}},\"recipientAccountId\":{\"321404829113\":7}},\"trigger\":{\"rule-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\",\"alert-type\":\"basic\",\"created-at\":\"2019-12-19 09:32:37.931000+00:00\",\"description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"risk\":3}}}"
| eval ErrorCode=spath(_raw,"detail.summary.Events.GetObject.Error Code")
| eval IP=spath(_raw,"detail.summary.IP")
| rex field=ErrorCode "(?<ErrorCode>\w+)"
| rex field=IP max_match=20 "(?<IP>(?<=\")[\w.]+)"
| fields - _*

Hi, @martinnepolean
The fields for the first question are missing, is this OK?

Re: Json field parsing

martinnepolean — Fri, 10 Jan 2020 11:53:22 GMT

I am looking to get all fields not only errorcode and IP. any help?

Re: Json field parsing

to4kawa — Fri, 10 Jan 2020 12:07:58 GMT

your sample doesn't have detail.summary events.createtags.isp
you say all fields. which?
Apparently the fields have been extracted.

Re: Json field parsing

kamlesh_vaghela — Fri, 10 Jan 2020 12:18:29 GMT

@martinnepolean

Can you please share your expected output from the sample you have shared?

Re: Json field parsing

martinnepolean — Wed, 30 Sep 2020 03:39:46 GMT

Hi Kamlesh,

One of the example to show how the field extraction happens, currently we are getting like below one

Detail.Summary.Events.GetObject.Error Code.AccessDenied=2

But we like to extract it as
Detail.Summary.Events.GetObject.Error Code = AccessDenied

and Below is the list of fields I am looking for from the above event shared.

version=0
id=2561455-c673-0hy6-673b-447895415
detail-type=Maciealert
source=aws.macie
account=123456789
time=2020-01-07T10:46:36Z
region=us-east-1
resources=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
detail.notification-type=ALERT_UPDATED
tags=Suspicious Access,Basic Alert
name=Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS"
severity=LOW
url=https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb
alert-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb
risk-score=3
updated-at=2020-01-07T10:46:36.136911
created-at=2020-01-07T00:46:35.139000+00:00
actor=321404829113:anonymous_principal
summary.Description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
IP=216.20.176.6,216.20.176.5,216.20.176.4,216.20.176.2
TimeRange="start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z","start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z","start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z","start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z","start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"
Location="us-east-1
Event Count=7
Events.GetObject.count=5
Events.GetObject.ISP=Company
Events.GetObject.Error Code=AccessDenied
Events.ListObjects.count=2
Events.ListObjects.ISP=Company
Events.ListObjects.Error Code=AccessDenied
recipientAccountId=321404829113
trigger.rule-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
trigger.alert-type=basic
trigger.created-at=2019-12-19 09:32:37.931000+00:00
trigger.description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
trigger.risk=3

Re: Json field parsing

to4kawa — Wed, 15 Jan 2020 12:48:52 GMT

spath, props.conf 's JSON extract is like this.
It is inevitable.