Hi everyone, I'm troubleshooting a Splunk Add-on for AWS CloudWatch Logs input that collects Amazon RDS Oracle audit logs. Environment: Splunk Add-on for AWS Boto3: 1.28.49 Botocore: 1.31.85 Python: 3.9.25 AWS Region: ap-southeast-1 CloudWatch Log Group: /aws/rds/instance/<RDS_INSTANCE>/audit RDS audit stream format: RDSDB_ora_*.xml The setup uses cross-account access: Splunk server
↓
Source/Source IAM Role
↓ AssumeRole
Target AWS IAM Role
↓
CloudWatch Logs The issue is that the existing IAM User works, while the new IAM Role does not, despite having the same relevant CloudWatch Logs permissions. Working IAM User The IAM User successfully performs: DescribeLogStreams
↓
GetLogEvents
↓
GetLogEvents ... CloudTrail shows GetLogEvents requests against the RDS audit log streams. For example, a successful request contains: logGroupName: /aws/rds/instance/<RDS_INSTANCE>/audit
logStreamName: RDSDB_ora_*.xml
startFromHead: true IAM Role The new target IAM Role is successfully assumed. CloudTrail identifies the caller as: arn:aws:sts::<ACCOUNT_ID>:assumed-role/<TARGET_ROLE>/splunk_ta_aws The role can successfully call: DescribeLogStreams However, CloudTrail shows repeated/paginated DescribeLogStreams calls with nextToken. The nextToken changes between requests, so it appears to be progressing through pagination. However, no GetLogEvents calls are observed for the IAM Role. The role therefore appears to be doing: AssumeRole
↓
DescribeLogStreams
↓
DescribeLogStreams + nextToken
↓
DescribeLogStreams + nextToken
↓
... but never reaches: GetLogEvents Additional information Both the IAM User and IAM Role inputs use the same start date and CloudWatch configuration. The working IAM User has been observed making GetLogEvents requests for historical RDS audit data, while the current date is significantly later. This raises the possibility that the newly created role-based input has a large historical backlog or a different checkpoint/state. We have considered: IAM permissions Permissions boundary AssumeRole session duration CloudWatch Logs resource policy Start date Pagination/checkpointing The role can already perform DescribeLogStreams, and the relevant CloudWatch Logs permissions appear equivalent to the working IAM User. Question Could this behaviour be related to the Splunk Add-on for AWS CloudWatch Logs checkpoint/state, particularly when using a newly created IAM Role/input compared with an existing IAM User/input? Specifically: Does a newly created CloudWatch Logs input start with a separate checkpoint and potentially have to process a large historical backlog? Could extensive DescribeLogStreams pagination prevent or significantly delay the first GetLogEvents request? Where can we check the CloudWatch Logs input's checkpoint or lastEventTimestamp in Splunk? Is there anything specific in the Splunk Add-on for AWS that could cause an IAM Role input to behave differently from an IAM User input when the AWS permissions are equivalent? What Splunk logs or troubleshooting steps would you recommend to determine why the role-based input does not progress from DescribeLogStreams to GetLogEvents? Any guidance would be greatly appreciated. Thank you!
... View more