splunk add on for aws doesnot report cloudwatch logs

[ssolipuram]

We are trying to ingest cloudwatch logs to splunk using splunk add-on for AWS. Some of the logs appear fine but there is a delay of more than 1 hour. The splunk server and forwarder are in the same time zone. And some of the logs dont even appear. Below is the error we are geting:

2018-05-18 17:14:32,803 level=ERROR pid=4348 tid=Thread-4 logger=splunk_ta_aws.modinputs.cloudwatch_logs.aws_cloudwatch_logs_data_loader pos=aws_cloudwatch_logs_data_loader.py:describe_cloudwatch_log_streams:73 | | message="Failure in describing cloudwatch logs streams due to throttling exception for log_group=app1/container, sleep=2.5481909735, reason=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch_logs/aws_cloudwatch_logs_data_loader.py", line 63, in describe_cloudwatch_log_streams
group_name, next_token=buf["nextToken"])
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/boto/logs/layer1.py", line 308, in describe_log_streams
body=json.dumps(params))
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/boto/logs/layer1.py", line 576, in make_request
body=json_body)
JSONResponseError: JSONResponseError: 400 Bad Request
{u'__type': u'ThrottlingException', u'message': u'Rate exceeded'}
"

Any help or suggestions are appreciated

[wendtb]

This error is due to throttling on AWS side. Unfortunately, there isn't a way to raise the exceeded throttling rate through AWS. The next best option is to utilize Kinesis Firehose ingestion into Splunk.

See this answer for throttling: https://answers.splunk.com/answers/482689/splunk-app-for-aws-getting-throttling-errors-for-a.html

Kinesis Info: https://www.splunk.com/blog/2017/11/29/ready-set-stream-with-the-kinesis-firehose-and-splunk-integra...
https://www.splunk.com/blog/2019/02/21/how-to-ingest-any-log-from-aws-cloudwatch-logs-via-firehose.h...
https://aws.amazon.com/blogs/big-data/power-data-ingestion-into-splunk-using-amazon-kinesis-data-fir...