CloudTrail Input - "The AWS account you have selected does not have sufficient permissions to access the data for this input.”

[n6BXGybt]

Splunk Version - 6.3.0
Splunk Build - aa7d4b1ccb80
Splunk App for AWS
App Version - 4.2.1

I have successfully setup 3 AWS accounts. However, for the fourth AWS account the CloudTrain input isn't working. I get the message, "The AWS account you have selected does not have sufficient permissions to access the data for this input.”

There seems to be an issue with SQS but I can't figure out what is wrong. AWS support said that my configuration is fine.

The logs below do not show any errors or clues:

splunk_ta_aws_cloudtrail_util.log
splunk_ta_aws_cloudtrail_main.log

[n6BXGybt]

I hate answering my own question but after a few exchanges with Splunk support, the problem was that the "cloudtrail:DescribeTrails" was left out in the documentation - it has now been updated to include the action:

http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions

As for the exactly why the accounts worked without that action after upgrading from 4.1 to 4.2 don't know but I updated those IAM policies just in case something breaks in a future update.

[Jeremiah]

What do the IAM roles look like for the three accounts you've setup so far? Are they identical? Also check the _internal logs (sourcetype=aws*) to see what specific permissions errors are appearing.

[n6BXGybt]

Thank you for your reply.

I am using IAM users, not IAM roles.

So, I have fixed the issue but I still don't understand why it is an issue.

Each IAM user in all accounts have this Inline Policy configured:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:GetQueueUrl",
                "sqs:SendMessage",
                "sqs:DeleteMessage",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "config:DeliverConfigSnapshot",
                "config:DescribeConfigRules",
                "config:DescribeConfigRuleEvaluationStatus",
                "config:GetComplianceDetailsByConfigRule",
                "config:GetComplianceSummaryByConfigRule",
                "config:DescribeDeliveryChannels",
                "autoscaling:Describe*",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "sns:Get*",
                "sns:List*",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "ec2:DescribeInstances",
                "ec2:DescribeReservedInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeRegions",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "rds:DescribeDBInstances",
                "cloudfront:ListDistributions",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeInstanceHealth",
                "inspector:Describe*",
                "inspector:List*",
                "kms:Decrypt",
                "kms:ReEncryptFrom"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

I ran an All time (real-time) query and then generated the error - unfortunately nothing is showing up in the Splunk Search & Reporting console for the problem account.

I decided to try looking at the Network section of my browser's Developer Tools and discovered the following error 500 Internal Server Error for this call:

https://172.20.4.10:8000/en-US/splunkd/__raw/servicesNS/nobody/splunk_app_aws/saas-aws/splunk_app_aw...

The response of that call is:

{"messages":[{"type":"ERROR","text":"\n In handler 'splunk_app_aws_aws_sqs': Unexpected error \"<class 'boto.exception.JSONResponseError'>\" from python handler: \"JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\".  See splunkd.log for more details."}]}

I ran sudo tail /opt/splunk/var/log/splunk/splunkd.log on the Splunk instance found this log:

07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 529, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_sqs_handler.py", line 57, in handleList\n    q_list = au.get_cloudtrail_sqs(proxy, region_name, aws_account)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 165, in get_cloudtrail_sqs\n    trails = conn.describe_trails()['trailList']\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 180, in describe_trails\n    body=json.dumps(params))\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 374, in make_request\n    body=json_body)\nJSONResponseError: JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::924074732483:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\n
07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'boto.exception.JSONResponseError'>" from python handler: "JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}".  See splunkd.log for more details.

So obviously, the problem is that cloudtrail:DescribeTrails is not part of the IAM policy - once I added this action I could add the account.

Okay....so....the other AWS accounts do not have this action in the IAM policies so why is the CloudTrail input working for them?

I performed a search for this error for my other AWS Accounts and it is only this account.

The only clue that I can give is that those accounts were already added to Splunk App for AWS before I upgraded to version 4.2.1.