Getting Data In

Why does AWS CloudWatch stop receiving events with the error "400 Bad Request {u'message': u'Rate exceeded', u'__type': u'ThrottlingException'}"?

sylim_splunk
Splunk Employee
Splunk Employee

We have configured large number of CloudWatch log groups as a separate input in our heavy forwarder. We have noticed that when pulling the logs from AWS instance, we are getting throttling exceptions for few of the log groups as mentioned below.

2016-10-25 10:33:18,164 ERROR pid=24573 tid=Thread-12 file=aws_cloudwatch_logs_data_loader.py:describe_cloudwatch_log_streams:74 | Failure in describing cloudwatch logs streams due to throttling exception for log_group=/okapi2/nprod/var/log/custom/vmstat, sleep=3.65761418489, reason=Traceback (most recent call last):
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/cloudwatch_logs_mod/aws_cloudwatch_logs_data_loader.py", line 64, in describe_cloudwatch_log_streams
group_name, next_token=buf["nextToken"])
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/boto/logs/layer1.py", line 308, in describe_log_streams
body=json.dumps(params))
File "/opt/app/splunk/etc/apps/Splunk_TA_aws/bin/boto/logs/layer1.py", line 576, in make_request
body=json_body)
JSONResponseError: JSONResponseError: 400 Bad Request
{u'message': u'Rate exceeded', u'__type': u'ThrottlingException'} 
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

The error message is from AWS which Splunk don't have any controls over , you can try to contact AWS and ask to increase API rate increase. Otherwise please switch to Kinesis input to work it around;

View solution in original post

jkat54
SplunkTrust
SplunkTrust

It says the reason for the error in the message:

Throttling Exception, Rate Exceeded.

So you've gone over your limit for downloading cloud watch logs. Yes AWS support can help you but you can probably find the solution quicker with a quick google search "increase cloud watch download limits"

In fact it was on the first link returned when I googled that: http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html

I think you need to request a rate increase on your "list metrics" or "describe alarms" limit. The lowest one is describe alarms so you might want to start there.

sylim_splunk
Splunk Employee
Splunk Employee

The error message is from AWS which Splunk don't have any controls over , you can try to contact AWS and ask to increase API rate increase. Otherwise please switch to Kinesis input to work it around;

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...