Getting Data In

Json field parsing

martinnepolean
Explorer

Hi,

We are getting the aws macie events as _json souretype, due to multiple loops there is a problem in fields extraction. I have give the screenshots below, red oval should be the field name and green oval should be valued.

for example the field name is detail.summary events.createtags.isp amazon and value is 436, but we need to field name till ISP and value to be amazon.

Please let me know how to get it gone through props and transforms

alt textalt text

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="{\"version\":\"0\",\"id\":\"2561455-c673-0hy6-673b-447895415\",\"detail-type\":\"Macie Alert\",\"source\":\"aws.macie\",\"account\":\"123456789\",\"time\":\"2020-01-07T10:46:36Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\"],\"detail\":{\"notification-type\":\"ALERT_UPDATED\",\"tags\":[\"Suspicious Access\",\"Basic Alert\"],\"name\":\"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS\",\"severity\":\"LOW\",\"url\":\"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%2F0c54ddb4cd37e6b8316ecdc1ba4ae3b7%2Falert%2F014f2161de2fffc59dd5d2cdf81a73fb\",\"alert-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb\",\"risk-score\":3,\"updated-at\":\"2020-01-07T10:46:36.136911\",\"created-at\":\"2020-01-07T00:46:35.139000+00:00\",\"actor\":\"321404829113:anonymous_principal\",\"summary\":{\"Description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"IP\":{\"216.20.176.6\":1,\"216.20.176.5\":1,\"216.20.176.4\":2,\"216.20.176.2\":3},\"Time Range\":[{\"count\":3,\"start\":\"2020-01-07T00:29:44Z\",\"end\":\"2020-01-07T00:29:47Z\"},{\"count\":1,\"start\":\"2020-01-07T10:06:11Z\",\"end\":\"2020-01-07T10:06:11Z\"},{\"count\":1,\"start\":\"2020-01-07T07:51:59Z\",\"end\":\"2020-01-07T07:51:59Z\"},{\"count\":1,\"start\":\"2020-01-07T10:19:18Z\",\"end\":\"2020-01-07T10:19:18Z\"},{\"count\":1,\"start\":\"2020-01-07T10:24:37Z\",\"end\":\"2020-01-07T10:24:37Z\"}],\"Record Count\":5,\"Location\":{\"us-east-1\":7},\"Event Count\":7,\"Events\":{\"GetObject\":{\"count\":5,\"ISP\":{\"Company\":5},\"Error Code\":{\"AccessDenied\":5}},\"ListObjects\":{\"count\":2,\"ISP\":{\"Company\":2},\"Error Code\":{\"AccessDenied\":2}}},\"recipientAccountId\":{\"321404829113\":7}},\"trigger\":{\"rule-arn\":\"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7\",\"alert-type\":\"basic\",\"created-at\":\"2019-12-19 09:32:37.931000+00:00\",\"description\":\"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content\",\"risk\":3}}}"
| eval ErrorCode=spath(_raw,"detail.summary.Events.GetObject.Error Code")
| eval IP=spath(_raw,"detail.summary.IP")
| rex field=ErrorCode "(?<ErrorCode>\w+)"
| rex field=IP max_match=20 "(?<IP>(?<=\")[\w.]+)"
| fields - _*

Hi, @martinnepolean
The fields for the first question are missing, is this OK?

0 Karma

martinnepolean
Explorer

I am looking to get all fields not only errorcode and IP. any help?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@martinnepolean

Can you please share your expected output from the sample you have shared?

0 Karma

martinnepolean
Explorer

Hi Kamlesh,

One of the example to show how the field extraction happens, currently we are getting like below one

Detail.Summary.Events.GetObject.Error Code.AccessDenied=2

But we like to extract it as
Detail.Summary.Events.GetObject.Error Code = AccessDenied

and Below is the list of fields I am looking for from the above event shared.

version=0
id=2561455-c673-0hy6-673b-447895415
detail-type=Maciealert
source=aws.macie
account=123456789
time=2020-01-07T10:46:36Z
region=us-east-1
resources=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
detail.notification-type=ALERT_UPDATED
tags=Suspicious Access,Basic Alert
name=Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS"
severity=LOW
url=https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%...
alert-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb
risk-score=3
updated-at=2020-01-07T10:46:36.136911
created-at=2020-01-07T00:46:35.139000+00:00
actor=321404829113:anonymous_principal
summary.Description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
IP=216.20.176.6,216.20.176.5,216.20.176.4,216.20.176.2
TimeRange="start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z","start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z","start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z","start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z","start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"
Location="us-east-1
Event Count=7
Events.GetObject.count=5
Events.GetObject.ISP=Company
Events.GetObject.Error Code=AccessDenied
Events.ListObjects.count=2
Events.ListObjects.ISP=Company
Events.ListObjects.Error Code=AccessDenied
recipientAccountId=321404829113
trigger.rule-arn=arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"
trigger.alert-type=basic
trigger.created-at=2019-12-19 09:32:37.931000+00:00
trigger.description=Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content
trigger.risk=3

0 Karma

to4kawa
Ultra Champion

spath, props.conf 's JSON extract is like this.
It is inevitable.

0 Karma

to4kawa
Ultra Champion

your sample doesn't have detail.summary events.createtags.isp
you say all fields. which?
Apparently the fields have been extracted.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@martinnepolean
Can you please share sample _raw events and expected output?

0 Karma

martinnepolean
Explorer

{"version":"0","id":"2561455-c673-0hy6-673b-447895415","detail-type":"Macie Alert","source":"aws.macie","account":"123456789","time":"2020-01-07T10:46:36Z","region":"us-east-1","resources":["arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7/alert/014f2161de2fffc59dd5d2cdf81a73fb","arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7"],"detail":{"notification-type":"ALERT_UPDATED","tags":["Suspicious Access","Basic Alert"],"name":"Access Denied to IAM user while attempting to get an AWS S3 Object from outside of AWS","severity":"LOW","url":"https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1%3A123456789%3Atrigger%... Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","IP":{"216.20.176.6":1,"216.20.176.5":1,"216.20.176.4":2,"216.20.176.2":3},"Time Range":[{"count":3,"start":"2020-01-07T00:29:44Z","end":"2020-01-07T00:29:47Z"},{"count":1,"start":"2020-01-07T10:06:11Z","end":"2020-01-07T10:06:11Z"},{"count":1,"start":"2020-01-07T07:51:59Z","end":"2020-01-07T07:51:59Z"},{"count":1,"start":"2020-01-07T10:19:18Z","end":"2020-01-07T10:19:18Z"},{"count":1,"start":"2020-01-07T10:24:37Z","end":"2020-01-07T10:24:37Z"}],"Record Count":5,"Location":{"us-east-1":7},"Event Count":7,"Events":{"GetObject":{"count":5,"ISP":{"Company":5},"Error Code":{"AccessDenied":5}},"ListObjects":{"count":2,"ISP":{"Company":2},"Error Code":{"AccessDenied":2}}},"recipientAccountId":{"321404829113":7}},"trigger":{"rule-arn":"arn:aws:macie:us-east-1:123456789:trigger/0c54ddb4cd37e6b8316ecdc1ba4ae3b7","alert-type":"basic","created-at":"2019-12-19 09:32:37.931000+00:00","description":"Access Denied error to IAM user while attempting to get an AWS S3 Object from and IP address outside of AWS. This could be an indication of attempted access to restricted content","risk":3}}}

I like to have error code as an field and AccessDenied as value which is not showing up now. and like to tag all IP addresses under IP field name.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...