I'm currently on Splunk Enterprise 6.3.0 and Splunk App for AWS 4.2.1 Are you using the Splunk Enterprise appliance from the AWS Marketplace? If so, I'm ready to update mine so if you don't mind sharing how you updated your AMI I can give it a shot and see if I run into the same problem. ... View more

I hate answering my own question but after a few exchanges with Splunk support, the problem was that the "cloudtrail:DescribeTrails" was left out in the documentation - it has now been updated to include the action: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions As for the exactly why the accounts worked without that action after upgrading from 4.1 to 4.2 don't know but I updated those IAM policies just in case something breaks in a future update. ... View more

Thank you for your reply. I am using IAM users, not IAM roles. So, I have fixed the issue but I still don't understand why it is an issue. Each IAM user in all accounts have this Inline Policy configured: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:GetQueueAttributes", "sqs:ListQueues", "sqs:ReceiveMessage", "sqs:GetQueueUrl", "sqs:SendMessage", "sqs:DeleteMessage", "s3:ListBucket", "s3:GetObject", "s3:GetBucketLocation", "s3:ListAllMyBuckets", "config:DeliverConfigSnapshot", "config:DescribeConfigRules", "config:DescribeConfigRuleEvaluationStatus", "config:GetComplianceDetailsByConfigRule", "config:GetComplianceSummaryByConfigRule", "config:DescribeDeliveryChannels", "autoscaling:Describe*", "cloudwatch:Describe*", "cloudwatch:Get*", "cloudwatch:List*", "sns:Get*", "sns:List*", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents", "ec2:DescribeInstances", "ec2:DescribeReservedInstances", "ec2:DescribeSnapshots", "ec2:DescribeRegions", "ec2:DescribeKeyPairs", "ec2:DescribeNetworkAcls", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVolumes", "ec2:DescribeVpcs", "rds:DescribeDBInstances", "cloudfront:ListDistributions", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeInstanceHealth", "inspector:Describe*", "inspector:List*", "kms:Decrypt", "kms:ReEncryptFrom" ], "Resource": [ "*" ] } ] } I ran an All time (real-time) query and then generated the error - unfortunately nothing is showing up in the Splunk Search & Reporting console for the problem account. I decided to try looking at the Network section of my browser's Developer Tools and discovered the following error 500 Internal Server Error for this call: https://172.20.4.10:8000/en-US/splunkd/__raw/servicesNS/nobody/splunk_app_aws/saas-aws/splunk_app_aws_aws_sqs?output_mode=json&aws_account=PROD+AWS+Account&aws_region=eu-west-1&aws_service=cloudtrail&target=127.0.0.1&count=-1&_=1469601370305 The response of that call is: {"messages":[{"type":"ERROR","text":"\n In handler 'splunk_app_aws_aws_sqs': Unexpected error \"<class 'boto.exception.JSONResponseError'>\" from python handler: \"JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\". See splunkd.log for more details."}]} I ran sudo tail /opt/splunk/var/log/splunk/splunkd.log on the Splunk instance found this log: 07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 529, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_sqs_handler.py", line 57, in handleList\n q_list = au.get_cloudtrail_sqs(proxy, region_name, aws_account)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 165, in get_cloudtrail_sqs\n trails = conn.describe_trails()['trailList']\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 180, in describe_trails\n body=json.dumps(params))\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/cloudtrail/layer1.py", line 374, in make_request\n body=json_body)\nJSONResponseError: JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::924074732483:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}\n 07-27-2016 06:58:43.932 +0000 ERROR AdminManagerExternal - Unexpected error "<class 'boto.exception.JSONResponseError'>" from python handler: "JSONResponseError: 400 Bad Request\n{u'__type': u'AccessDeniedException', u'Message': u'User: arn:aws:iam::XXXXXXXXXXXX:user/PROD-SPLUNKAPP-USER is not authorized to perform: cloudtrail:DescribeTrails'}". See splunkd.log for more details. So obviously, the problem is that cloudtrail:DescribeTrails is not part of the IAM policy - once I added this action I could add the account. Okay....so....the other AWS accounts do not have this action in the IAM policies so why is the CloudTrail input working for them? I performed a search for this error for my other AWS Accounts and it is only this account. The only clue that I can give is that those accounts were already added to Splunk App for AWS before I upgraded to version 4.2.1. ... View more

No reply so I simply use crontabs to update the security packages only: yum update --security -y ... View more

I'm not sure about question 1) however for question 2) I suspect that you don't need an indexer for the AWS specific logs as there are no Sizing and Performance considerations. Reference: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/Sizingandcost In my case, I have four AWS accounts (DEV, QA, UAT, and PROD). Since assume role is not supported, I have an IAM user account in each AWS account - DEV-SPLUNKAPP-USER for example. This user has in inline policy configured as below and per the Splunk documentation for using one policy for all inputs: Reference: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:GetQueueAttributes", "sqs:ListQueues", "sqs:ReceiveMessage", "sqs:GetQueueUrl", "sqs:SendMessage", "sqs:DeleteMessage", "s3:ListBucket", "s3:GetObject", "s3:GetBucketLocation", "s3:ListAllMyBuckets", "config:DeliverConfigSnapshot", "config:DescribeConfigRules", "config:DescribeConfigRuleEvaluationStatus", "config:GetComplianceDetailsByConfigRule", "config:GetComplianceSummaryByConfigRule", "config:DescribeDeliveryChannels", "cloudtrail:DescribeTrails", "autoscaling:Describe*", "cloudwatch:Describe*", "cloudwatch:Get*", "cloudwatch:List*", "sns:Get*", "sns:List*", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents", "ec2:DescribeInstances", "ec2:DescribeReservedInstances", "ec2:DescribeSnapshots", "ec2:DescribeRegions", "ec2:DescribeKeyPairs", "ec2:DescribeNetworkAcls", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVolumes", "ec2:DescribeVpcs", "rds:DescribeDBInstances", "cloudfront:ListDistributions", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeInstanceHealth", "inspector:Describe*", "inspector:List*" ], "Resource": [ "*" ] } ] } In my case I also created a separate index in Splunk (Settings --> Indexes) for each AWS service. (eg. config_logs, cloudwatch_logs, etc.) ... View more

Thanks for your reply. When I look at the documentation below, I don't see any mention of using a role on the Splunk instance: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions The reason why I have this role applied to my Splunk instance is because we assign IAM roles to all instances since IAM roles cannot retroactively be added to instances. Not only is it a best practice but I do this for cases where I need to copy a file to S3 - for example the Splunk diagnostics file and it's easy to do with a temporary IAM policy applied to the IAM role. Unfortunately when I wrote my question for some reason I thought it was a stale account but as you mentioned, Splunk automatically detected the role and I guess I didn't notice before. This IAM role is for our own general AWS infrastructure management and not related to Splunk specifically. So it looks like I asked the wrong question. 😞 Apologies for this. The take away is that IAM roles applied to Splunk instances will automatically be detected and displayed in the Accounts List of the Splunk App for AWS and cannot be deleted, correct? ... View more

I tried removing from the add-on configuration UI but unless I'm looking at the wrong spot, there doesn't seem to be a way to remove. I ran a tail while trying to remove and found this log: sudo tail -F /opt/splunk/var/log/splunk/splunk_ta_aws_util.log Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_rh_settings_account.py", line 103, in handleRemove km.delete_accesskey(self.callerArgs.id) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/taaws/aws_accesskeys.py", line 123, in delete_accesskey self._cred_mgr.delete(c.key_id, c.name, self.namespace, self.owner) File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/taaws/credentials.py", line 110, in delete response, content = splunk.rest.simpleRequest(cred_id, method='DELETE', raiseAllErrors=True, sessionKey=self._sessionKey) File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 529, in simpleRequest raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body)) ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/-/Splunk_TA_aws/storage/passwords/M1-LOGGING-ROLE%3AASIAJFJEIOJ4E6T2JQVA%3A; [{'text': "\n In handler 'passwords': Delete user failed, cannot find the credential information with id : credential:M1-LOGGING-ROLE:ASIAJFJEIOJ4E6T2JQVA:", 'code': None, 'type': 'ERROR'}] What I don't understand is that the account I'm trying to remove is actually an IAM role so it shouldn't be there in the first place. Looking at the log above it seems that the script is looking for the secret key for the access key ASIAJFJEIOJ4E6T2JQVA. However, since this is a role, there isn't one assigned. The EC2 role that is referenced is actually applied to the Splunk Enterprise instance and only has a policy assigned to monitor memory metrics and send to CloudWatch. ... View more