I'm trying to set up CloudTrail log ingestion using the AWS splunk addon and using IAM roles.
Details:
Splunk 7.3.3
Splunk Add-on for AWS 4.6.0
Splunk App for AWS 5.1.0
We have an account 123 with key id xyz, set to Region Category Global.
This was configured as an SQS-based S3 input, with no asume role, us-west (oregon)region , with a sqs queue created in AWS
Batch size is 10, S3 file decoder is CloudTrail, sourcetype is aws:cloudtrail, interval 300 sec
This account works just fine and we are getting logs in our index.
We also configured a IAM role for another account 345, that uses a Role ARN of arn:aws:iam:345:role/some_iam_AWS_role
We created another input with the same settings but with an assume role (eg with put a different name for the input, same 123 account with assume role of 345).
This input doesn`t download any logs in the index and we have the following details:
1.Log error extracted form splunk_ta_aws_aws_sqs_based_s3_input-name-345-1.log ($SPLUNK_HOME/var/log/splunk/splunk_ta_aws)
(~ 3 pairs / second ), log files are rotated fast.
Previously, we had:
ClientError: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
2019-12-11 13:46:47,101 level=ERROR pid=19928 tid=Thread-1 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_parse:318
This changed after some config tries on the AWS side to the one below:
2019-12-11 19:51:09,232 level=ERROR pid=24254 tid=Thread-9 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_download:292 | start_time=1576088583 datainput="input-name-345-1", message_id="1234567-1234-1234-1234-1234567890abc" created=1576093868.94 job_id=abcdef-abcd-abbd-ssdd-2233aaabbccddd ttl=30 | message="Failed to download file."
2019-12-11 19:51:09,260 level=CRITICAL pid=24254 tid=Thread-9 logger=splunk_ta_aws.modinputs.sqs_based_s3.handler pos=handler.py:_process:268 | start_time=1576088583 datainput="input-name-345-1", message_id="1234567-1234-1234-1234-1234567890abc" created=1576093868.94 job_id=abcdef-abcd-abbd-ssdd-2233aaabbcc ttl=30 | message="An error occurred while processing the message."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 256, in _process
headers = self._download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 290, in _download
return self._s3_agent.download(record, cache, session)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/sqs_based_s3/handler.py", line 418, in download
return bucket.transfer(s3, key, fileobj, **condition)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/common/s3.py", line 73, in transfer
headers = client.head_object(Bucket=bucket, Key=key, **kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/client.py", line 324, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/client.py", line 622, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
2.looking at the _internal index, i can see logs with message=request role credentials every hour, and the same Failed to download file. messages
3.These assumed role inputs never worked. We get ~ 60k req / hour that fail in splunk.
4.The assume account has the following permissions
"s3:GetAccelerateConfiguration",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketLogging",
"s3:GetBucketTagging",
"s3:GetLifecycleConfiguration",
"s3:GetObject",
"s3:ListAllMyBuckets",
"s3:ListBucket",
"sns:Get*",
Any input is appreciated, as well as any questions for additional data needed.
Hello, I have the same error, Could you tell me if you were able to solve it?
policy:
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:Describe*",
"cloudfront:ListDistributions",
"cloudwatch:Describe*",
"cloudwatch:Get*",
"cloudwatch:List*",
"config:DeliverConfigSnapshot",
"config:DescribeConfigRuleEvaluationStatus",
"config:DescribeConfigRules",
"config:GetComplianceDetailsByConfigRule",
"config:GetComplianceSummaryByConfigRule",
"ec2:DescribeAddresses",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeKeyPairs",
"ec2:DescribeNetworkAcls",
"ec2:DescribeRegions",
"ec2:DescribeReservedInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSnapshots",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:DescribeVpcs",
"elasticloadbalancing:DescribeInstanceHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"iam:GetAccessKeyLastUsed",
"iam:GetAccountPasswordPolicy",
"iam:GetUser",
"iam:ListAccessKeys",
"iam:ListUsers",
"inspector:Describe*",
"inspector:List*",
"kinesis:DescribeStream",
"kinesis:Get*",
"kinesis:ListStreams",
"kms:Decrypt",
"lambda:ListFunctions",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:GetLogEvents",
"rds:DescribeDBInstances",
"s3:GetAccelerateConfiguration",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketLogging",
"s3:GetBucketTagging",
"s3:GetLifecycleConfiguration",
"s3:GetObject",
"s3:ListAllMyBuckets",
"s3:ListBucket",
"sns:Get*",
"sns:List*",
"sns:Publish",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ListQueues",
"sqs:ReceiveMessage",
"sqs:SendMessage",
"sts:AssumeRole"
],
"Resource": [
"*"
]
}
]
}
Did you fix the issue? Would like to know what you did other than the blog posted?
I am running into this issue as well, one recommended solution is using Lambda per my research saying its a limitation of the SNS queues. Referencing the following article:
Were you able to resolve this? If so, what approach did you take?