Getting Data In

Why does AWS CloudWatch stop receiving events with the error "400 Bad Request {u'message': u'Rate exceeded', u'__type': u'ThrottlingException'}"?

sylim_splunk
Splunk Employee
Splunk Employee

We have configured large number of CloudWatch log groups as a separate input in our heavy forwarder. We have noticed that when pulling the logs from AWS instance, we are getting throttling exceptions for few of the log groups as mentioned below.

2016-10-25 10:33:18,164 ERROR pid=24573 tid=Thread-12 file=aws_cloudwatch_logs_data_loader.py:describe_cloudwatch_log_streams:74 | Failure in describing cloudwatch logs streams due to throttling exception for log_group=/okapi2/nprod/var/log/custom/vmstat, sleep=3.65761418489, reason=Traceback (most recent call last):
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/cloudwatch_logs_mod/aws_cloudwatch_logs_data_loader.py", line 64, in describe_cloudwatch_log_streams
group_name, next_token=buf["nextToken"])
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/boto/logs/layer1.py", line 308, in describe_log_streams
body=json.dumps(params))
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/boto/logs/layer1.py", line 576, in make_request
body=json_body)
JSONResponseError: JSONResponseError: 400 Bad Request
{u'message': u'Rate exceeded', u'__type': u'ThrottlingException'} 
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

The error message is from AWS which Splunk don't have any controls over , you can try to contact AWS and ask to increase API rate increase. Otherwise please switch to Kinesis input to work it around;

View solution in original post

jkat54
SplunkTrust
SplunkTrust

It says the reason for the error in the message:

Throttling Exception, Rate Exceeded.

So you've gone over your limit for downloading cloud watch logs. Yes AWS support can help you but you can probably find the solution quicker with a quick google search "increase cloud watch download limits"

In fact it was on the first link returned when I googled that: http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html

I think you need to request a rate increase on your "list metrics" or "describe alarms" limit. The lowest one is describe alarms so you might want to start there.

sylim_splunk
Splunk Employee
Splunk Employee

The error message is from AWS which Splunk don't have any controls over , you can try to contact AWS and ask to increase API rate increase. Otherwise please switch to Kinesis input to work it around;

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...