All Apps and Add-ons

CloudTrail Input - "The AWS account you have selected does not have sufficient permissions to access the data for this input.”

n6BXGybt
Path Finder

Splunk Version - 6.3.0
Splunk Build - aa7d4b1ccb80
Splunk App for AWS
App Version - 4.2.1

I have successfully setup 3 AWS accounts. However, for the fourth AWS account the CloudTrain input isn't working. I get the message, "The AWS account you have selected does not have sufficient permissions to access the data for this input.”

There seems to be an issue with SQS but I can't figure out what is wrong. AWS support said that my configuration is fine.

The logs below do not show any errors or clues:
alt text
splunk_ta_aws_cloudtrail_util.log
splunk_ta_aws_cloudtrail_main.log

alt text

0 Karma

n6BXGybt
Path Finder

I hate answering my own question but after a few exchanges with Splunk support, the problem was that the "cloudtrail:DescribeTrails" was left out in the documentation - it has now been updated to include the action:

http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions

As for the exactly why the accounts worked without that action after upgrading from 4.1 to 4.2 don't know but I updated those IAM policies just in case something breaks in a future update.

Jeremiah
Motivator

What do the IAM roles look like for the three accounts you've setup so far? Are they identical? Also check the _internal logs (sourcetype=aws*) to see what specific permissions errors are appearing.

0 Karma

n6BXGybt
Path Finder

Thank you for your reply.

I am using IAM users, not IAM roles.

So, I have fixed the issue but I still don't understand why it is an issue.

Each IAM user in all accounts have this Inline Policy configured:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:GetQueueUrl",
                "sqs:SendMessage",
                "sqs:DeleteMessage",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "config:DeliverConfigSnapshot",
                "config:DescribeConfigRules",
                "config:DescribeConfigRuleEvaluationStatus",
                "config:GetComplianceDetailsByConfigRule",
                "config:GetComplianceSummaryByConfigRule",
                "config:DescribeDeliveryChannels",
                "autoscaling:Describe*",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "sns:Get*",
                "sns:List*",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "ec2:DescribeInstances",
                "ec2:DescribeReservedInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeRegions",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "rds:DescribeDBInstances",
                "cloudfront:ListDistributions",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeInstanceHealth",
                "inspector:Describe*",
                "inspector:List*",
                "kms:Decrypt",
                "kms:ReEncryptFrom"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

I ran an All time (real-time) query and then generated the error - unfortunately nothing is showing up in the Splunk Search & Reporting console for the problem account.

I decided to try looking at the Network section of my browser's Developer Tools and discovered the following error 500 Internal Server Error for this call:

https://172.20.4.10:8000/en-US/splunkd/__raw/servicesNS/nobody/splunk_app_aws/saas-aws/splunk_app_aw...

The response of that call is:

{"messages":[{"type":"ERROR","text":"\n In handler 'splunk_app_aws_aws_sqs': Unexpected error \"<class 'boto.exception.JSONResponseError'>\" from python handler: \"JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\".  See splunkd.log for more details."}]}

I ran sudo tail /opt/splunk/var/log/splunk/splunkd.log on the Splunk instance found this log:

07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 529, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_sqs_handler.py", line 57, in handleList\n    q_list = au.get_cloudtrail_sqs(proxy, region_name, aws_account)\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 165, in get_cloudtrail_sqs\n    trails = conn.describe_trails()['trailList']\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 180, in describe_trails\n    body=json.dumps(params))\n  File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 374, in make_request\n    body=json_body)\nJSONResponseError: JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::924074732483:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\n
07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'boto.exception.JSONResponseError'>" from python handler: "JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}".  See splunkd.log for more details.

So obviously, the problem is that cloudtrail:DescribeTrails is not part of the IAM policy - once I added this action I could add the account.

Okay....so....the other AWS accounts do not have this action in the IAM policies so why is the CloudTrail input working for them?

I performed a search for this error for my other AWS Accounts and it is only this account.

The only clue that I can give is that those accounts were already added to Splunk App for AWS before I upgraded to version 4.2.1.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...