topic Cannot break AWS Cloudtrail events in Splunk Search

Cannot break AWS Cloudtrail events

colny — Wed, 30 Sep 2020 04:30:22 GMT

Hi,
The cloudtrail logs in splunk come in without proper event break; I only got it to recognize the first event's timestamp.
This is problem because each 'Records' do contain a large number of separate events with its own timestamp.
Each event is a json block starting with eventVersion.

Here is an anonymized sample :
{
"Records": [
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AAAAAAAAAAAAAAAA12345",
"arn": "arn:aws:iam::999999999999:user/S3_ContentProvider",
"accountId": "999999999999",
"accessKeyId": "BBBBBBBBBBBBBBB12345",
"userName": "S3_ContentProvider"
},
"eventTime": "2020-03-05T04:16:50Z",
"eventSource": "sns.amazonaws.com",
"eventName": "ListTopics",
"awsRegion": "us-east-1",
"sourceIPAddress": "10.10.10.10",
"userAgent": "aws-sdk-java/1.11.192 Linux/3.10.0-693.21.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 exec-env/AWS_ECS_EC2",
"requestParameters": null,
"responseElements": null,
"requestID": "0000000a-000a-000a-000a-00000000000a",
"eventID": "0000000a-000a-000a-000a-00000000000b",
"eventType": "AwsApiCall",
"recipientAccountId": "999999999999"
},
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AAAAAAAAAAAAAAAA12345",
"arn": "arn:aws:iam::999999999999:user/S3_ContentProvider",
"accountId": "999999999999",
"accessKeyId": "BBBBBBBBBBBBBBB12345",
"userName": "S3_ContentProvider"
},
"eventTime": "2020-03-05T04:17:04Z",
"eventSource": "sns.amazonaws.com",
"eventName": "ListTopics",
"awsRegion": "us-east-1",
"sourceIPAddress": "10.10.10.11",
"userAgent": "aws-sdk-java/1.11.192 Linux/3.10.0-693.21.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 exec-env/AWS_ECS_EC2",
"requestParameters": null,
"responseElements": null,
"requestID": "0000000a-000a-000a-000a-00000000000a",
"eventID": "0000000a-000a-000a-000a-00000000000b",
"eventType": "AwsApiCall",
"recipientAccountId": "999999999999"
}
]
}

Thanks.

Re: Cannot break AWS Cloudtrail events

to4kawa — Fri, 06 Mar 2020 09:48:09 GMT

what's your props.conf ?

Re: Cannot break AWS Cloudtrail events

colny — Fri, 06 Mar 2020 10:57:00 GMT

Found the answer! https://stackoverflow.com/questions/60110812/splunk-props-config-for-aws-cloudtrail-json-logs

Re: Cannot break AWS Cloudtrail events

to4kawa — Fri, 06 Mar 2020 11:25:00 GMT

|makeresults
| eval _raw="{
\"Records\": [
{
\"eventVersion\": \"1.05\",
\"userIdentity\": {
\"type\": \"IAMUser\",
\"principalId\": \"AAAAAAAAAAAAAAAA12345\",
\"arn\": \"arn:aws:iam::999999999999:user/S3_ContentProvider\",
\"accountId\": \"999999999999\",
\"accessKeyId\": \"BBBBBBBBBBBBBBB12345\",
\"userName\": \"S3_ContentProvider\"
},
\"eventTime\": \"2020-03-05T04:16:50Z\",
\"eventSource\": \"sns.amazonaws.com\",
\"eventName\": \"ListTopics\",
\"awsRegion\": \"us-east-1\",
\"sourceIPAddress\": \"10.10.10.10\",
\"userAgent\": \"aws-sdk-java/1.11.192 Linux/3.10.0-693.21.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 exec-env/AWS_ECS_EC2\",
\"requestParameters\": null,
\"responseElements\": null,
\"requestID\": \"0000000a-000a-000a-000a-00000000000a\",
\"eventID\": \"0000000a-000a-000a-000a-00000000000b\",
\"eventType\": \"AwsApiCall\",
\"recipientAccountId\": \"999999999999\"
},
{
\"eventVersion\": \"1.05\",
\"userIdentity\": {
\"type\": \"IAMUser\",
\"principalId\": \"AAAAAAAAAAAAAAAA12345\",
\"arn\": \"arn:aws:iam::999999999999:user/S3_ContentProvider\",
\"accountId\": \"999999999999\",
\"accessKeyId\": \"BBBBBBBBBBBBBBB12345\",
\"userName\": \"S3_ContentProvider\"
},
\"eventTime\": \"2020-03-05T04:17:04Z\",
\"eventSource\": \"sns.amazonaws.com\",
\"eventName\": \"ListTopics\",
\"awsRegion\": \"us-east-1\",
\"sourceIPAddress\": \"10.10.10.11\",
\"userAgent\": \"aws-sdk-java/1.11.192 Linux/3.10.0-693.21.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 exec-env/AWS_ECS_EC2\",
\"requestParameters\": null,
\"responseElements\": null,
\"requestID\": \"0000000a-000a-000a-000a-00000000000a\",
\"eventID\": \"0000000a-000a-000a-000a-00000000000b\",
\"eventType\": \"AwsApiCall\",
\"recipientAccountId\": \"999999999999\"
}
]
}"
| rex mode=sed "s/,(?=\s*\{\s*\"eventVersion\")/#/g"

you need LINE_BREAKER=,(?=\s*\{\s*\"eventVersion\")
and link's SEDCMD

Re: Cannot break AWS Cloudtrail events

sbattista — Thu, 18 Nov 2021 20:25:34 GMT

This worked for me, thank you!