All Apps and Add-ons

splunk add on for aws doesnot report cloudwatch logs

Explorer

We are trying to ingest cloudwatch logs to splunk using splunk add-on for AWS. Some of the logs appear fine but there is a delay of more than 1 hour. The splunk server and forwarder are in the same time zone. And some of the logs dont even appear. Below is the error we are geting:

2018-05-18 17:14:32,803 level=ERROR pid=4348 tid=Thread-4 logger=splunk_ta_aws.modinputs.cloudwatch_logs.aws_cloudwatch_logs_data_loader pos=aws_cloudwatch_logs_data_loader.py:describe_cloudwatch_log_streams:73 | | message="Failure in describing cloudwatch logs streams due to throttling exception for log_group=app1/container, sleep=2.5481909735, reason=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/cloudwatch_logs/aws_cloudwatch_logs_data_loader.py", line 63, in describe_cloudwatch_log_streams
group_name, next_token=buf["nextToken"])
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/boto/logs/layer1.py", line 308, in describe_log_streams
body=json.dumps(params))
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/boto/logs/layer1.py", line 576, in make_request
body=json_body)
JSONResponseError: JSONResponseError: 400 Bad Request
{u'__type': u'ThrottlingException', u'message': u'Rate exceeded'}
"

Any help or suggestions are appreciated

0 Karma

Path Finder
0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!