Archive

AWS SQS error message="Failed to download file." while using IAM Roles

Engager

I'm trying to set up CloudTrail log ingestion using the AWS splunk addon and using IAM roles.

Details:
Splunk 7.3.3
Splunk Add-on for AWS 4.6.0
Splunk App for AWS 5.1.0

We have an account 123 with key id xyz, set to Region Category Global.

This was configured as an SQS-based S3 input, with no asume role, us-west (oregon)region , with a sqs queue created in AWS
Batch size is 10, S3 file decoder is CloudTrail, sourcetype is aws:cloudtrail, interval 300 sec

This account works just fine and we are getting logs in our index.

We also configured a IAM role for another account 345, that uses a Role ARN of arn:aws:iam:345:role/some_iam_AWS_role
We created another input with the same settings but with an assume role (eg with put a different name for the input, same 123 account with assume role of 345).

This input doesn`t download any logs in the index and we have the following details:

1.Log error extracted form splunk_ta_aws_aws_sqs_based_s3_input-name-345-1.log ($SPLUNK_HOME/var/log/splunk/splunk_ta_aws)

(~ 3 pairs / second ), log files are rotated fast.

Previously, we had:
ClientError: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
2019-12-11 13:46:47,101 level=ERROR pid=19928 tid=Thread-1 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_parse:318

This changed after some config tries on the AWS side to the one below:

2019-12-11 19:51:09,232 level=ERROR pid=24254 tid=Thread-9 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_download:292 | start_time=1576088583 datainput="input-name-345-1", message_id="1234567-1234-1234-1234-1234567890abc" created=1576093868.94 job_id=abcdef-abcd-abbd-ssdd-2233aaabbccddd ttl=30 | message="Failed to download file."
2019-12-11 19:51:09,260 level=CRITICAL pid=24254 tid=Thread-9 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:268 | start_time=1576088583 datainput="input-name-345-1", message_id="1234567-1234-1234-1234-1234567890abc" created=1576093868.94 job_id=abcdef-abcd-abbd-ssdd-2233aaabbcc ttl=30 | message="An error occurred while processing the message."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 256, in _process
headers = self._download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 290, in _download
return self._s3_agent.download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 418, in download
return bucket.transfer(s3, key, fileobj, **condition)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/s3.py", line 73, in transfer
headers = client.head_object(Bucket=bucket, Key=key, **kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/client.py", line 324, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/client.py", line 622, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

2.looking at the _internal index, i can see logs with message=request role credentials every hour, and the same Failed to download file. messages

3.These assumed role inputs never worked. We get ~ 60k req / hour that fail in splunk.

4.The assume account has the following permissions
"s3:GetAccelerateConfiguration",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketLogging",
"s3:GetBucketTagging",
"s3:GetLifecycleConfiguration",
"s3:GetObject",
"s3:ListAllMyBuckets",
"s3:ListBucket",
"sns:Get*",

Any input is appreciated, as well as any questions for additional data needed.

Tags (1)
0 Karma

Engager
  • policy:

    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": [
    "autoscaling:Describe*",
    "cloudfront:ListDistributions",
    "cloudwatch:Describe*",
    "cloudwatch:Get*",
    "cloudwatch:List*",
    "config:DeliverConfigSnapshot",
    "config:DescribeConfigRuleEvaluationStatus",
    "config:DescribeConfigRules",
    "config:GetComplianceDetailsByConfigRule",
    "config:GetComplianceSummaryByConfigRule",
    "ec2:DescribeAddresses",
    "ec2:DescribeImages",
    "ec2:DescribeInstances",
    "ec2:DescribeKeyPairs",
    "ec2:DescribeNetworkAcls",
    "ec2:DescribeRegions",
    "ec2:DescribeReservedInstances",
    "ec2:DescribeSecurityGroups",
    "ec2:DescribeSnapshots",
    "ec2:DescribeSubnets",
    "ec2:DescribeVolumes",
    "ec2:DescribeVpcs",
    "elasticloadbalancing:DescribeInstanceHealth",
    "elasticloadbalancing:DescribeListeners",
    "elasticloadbalancing:DescribeLoadBalancers",
    "elasticloadbalancing:DescribeTags",
    "elasticloadbalancing:DescribeTargetGroups",
    "elasticloadbalancing:DescribeTargetHealth",
    "iam:GetAccessKeyLastUsed",
    "iam:GetAccountPasswordPolicy",
    "iam:GetUser",
    "iam:ListAccessKeys",
    "iam:ListUsers",
    "inspector:Describe*",
    "inspector:List*",
    "kinesis:DescribeStream",
    "kinesis:Get*",
    "kinesis:ListStreams",
    "kms:Decrypt",
    "lambda:ListFunctions",
    "logs:DescribeLogGroups",
    "logs:DescribeLogStreams",
    "logs:GetLogEvents",
    "rds:DescribeDBInstances",
    "s3:GetAccelerateConfiguration",
    "s3:GetBucketCORS",
    "s3:GetBucketLocation",
    "s3:GetBucketLogging",
    "s3:GetBucketTagging",
    "s3:GetLifecycleConfiguration",
    "s3:GetObject",
    "s3:ListAllMyBuckets",
    "s3:ListBucket",
    "sns:Get*",
    "sns:List*",
    "sns:Publish",
    "sqs:DeleteMessage",
    "sqs:GetQueueAttributes",
    "sqs:GetQueueUrl",
    "sqs:ListQueues",
    "sqs:ReceiveMessage",
    "sqs:SendMessage",
    "sts:AssumeRole"
    ],
    "Resource": [
    "*"
    ]
    }
    ]
    }

0 Karma