IAM Activity, User Activity, CloudTrail API Activity Not Displaying

[asbetsplunk]

Splunk Add-on for AWS: 3.0.0
Splunk App for AWS: 4.1.1

I am trying out Splunk and the Splunk App for AWS. I have configured most of the inputs and most everything seems to be working but I'm experiencing three issues.

1. IAM Activity is empty even though I have created IAM events. (i.e. Update policy, modify KMS policy, etc.) - UPDATE: This working now. Just needed to be patient.
2. User Activity is showing an insane number of users when there are actually only 2. (screenshot included)
3. The CloudTrail API calls in the AWS Console are not showing up in the Splunk App for AWS. - UPDATE: This working now. Just needed to be patient.

It might be worth noting that my CloudTrail logs are encrypted with KMS. However, I have already observed and fixed the error "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4" by adding S3_USE_SIGV4 = True to the splunk-launch.conf file.

[czhang_splunk]

Hi asbet,
For your question 2, could you please just click on the number (55) to drilldown to the search to see what 55 users are?

Thanks

[asbetsplunk]

czhang,

Thanks for your reply.

The count now is 569 with one error. It looks like the KMS STS token assume role actions being counted as "users". I have provided a link to the screenshot - all of those STS entries are from the KMS service and the content of those are like what is pasted below:

{"eventName": "GenerateDataKey", "sourceIPAddress": "internal.amazonaws.com", "eventTime": "2016-04-21T21:08:51Z", "requestID": "3f520fc0-0805-11e6-9208-7f0131d283de", "resources": [{"accountId": "XXXXXXXXXXX", "ARN": "arn:aws:kms:eu-west-1:XXXXXXXXXXX:key/58860848-99ce-4248-b974-c18e3ad8a48e"}], "userAgent": "internal.amazonaws.com", "eventVersion": "1.04", "userIdentity": {"invokedBy": "internal.amazonaws.com", "type": "AssumedRole", "accountId": "035351147821", "sessionContext": {"attributes": {"creationDate": "2016-04-21T20:40:31Z", "mfaAuthenticated": "false"}, "sessionIssuer": {"userName": "AWSCloudTrail", "arn": "arn:aws:iam::035351147821:role/AWSCloudTrail", "type": "Role", "accountId": "035351147821", "principalId": "AROAIMYTXX4VMR4TEHMIU"}}, "principalId": "AROAIMYTXX4VMR4TEHMIU:i-00283eca92f510992", "accessKeyId": "XXXXXXXXXXX", "arn": "arn:aws:sts::035351147821:assumed-role/AWSCloudTrail/i-00283eca92f510992"}, "sharedEventID": "e9174339-8a97-4742-ab32-1f4c282042f0", "readOnly": true, "awsRegion": "eu-west-1", "eventType": "AwsApiCall", "responseElements": null, "recipientAccountId": "XXXXXXXXXXX", "eventID": "d532a32d-992b-4345-8ff6-f5d502d526d0", "eventSource": "kms.amazonaws.com", "requestParameters": {"encryptionContext": {"aws:s3:arn": "arn:aws:s3:::dev-m1-cloudtrail-logs/AWSLogs/XXXXXXXXXXX/CloudTrail/eu-central-1/2016/04/21/XXXXXXXXXXX_CloudTrail_eu-central-1_20160421T2110Z_dmfwwDvr7DETuCkR.json.gz", "aws:cloudtrail:arn": "arn:aws:cloudtrail:eu-west-1:XXXXXXXXXXX:trail/dev-m1-cloudtrail"}, "keyId": "arn:aws:kms:eu-west-1:XXXXXXXXXXX:key/58860848-99ce-4248-b974-c18e3ad8a48e", "keySpec": "AES_256"}}

[asbetsplunk]

I'm at well over 1,280 "users" now. "Decrypt" and "Generate Datakey" actions are being counted as users it seems.