AWS Inspector Rate Limited - ListFindings

[ejharts2015]

I'm posting this here as I wasn't seeing many others reporting this issue nor resolution and hope it saves someone else a lot of headache in solving it.

We setup AWS Inspector for our four AWS accounts to check for scan results once a day and we were seeing odd Rate Limit Exceeded Errors and only a limited number of results were getting pulled in.

2021-03-17 14:17:31,524 level=ERROR pid=32092 tid=Thread-6 logger=splunk_ta_aws.modinputs.inspector.aws_inspector_data_loader pos=aws_inspector_data_loader.py:__call__:307 | | message="Failed to collect inspector findings for region=us-east-1, datainput=Corp - Inspector, error=Traceback (most recent call last):
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 302, in __call__
self._do_indexing()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 284, in _do_indexing
AWSInspectorFindingsDataLoader(self._config, self._cli, account_id).run()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 176, in run
self._schedule()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 191, in _schedule
arns = self._list_findings_by_time_window(begin, end)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 249, in _list_findings_by_time_window
response = self._cli.list_findings(**params)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/client.py", line 586, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListFindings operation (reached max retries: 4): Rate exceeded
"

At once a day, it seems like we should not be hitting this ThrottlingException.

[ejharts2015 · Accepted solution]

This issue is resolved by increasing the max number of results returned per API request. By default (AWS Documentation) the ListFindings maxResults returns 10 results per call. Increasing this small value to a higher one resolved our issue.

Python Script to modify:

/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py

And the function to modify:

 def _list_findings_by_time_window(self, begin, end):
    # boto3 do not accept unix timestamp on windows
    # cast to datetime by hand
    begin = datetime.datetime.utcfromtimestamp(begin)
    end = datetime.datetime.utcfromtimestamp(end)
    params = {
        "filter": {
            'creationTimeRange': {
                'beginDate': begin,
                'endDate': end
            }
        },
        "maxResults": 500
    }

Resulted in far less API calls and no further rate-limiting. I've put in an enhancement request for this function's defaults to be modified. Hope this helps!

API calls before/after change.