All Apps and Add-ons

AWS Inspector Rate Limited - ListFindings

ejharts2015
Communicator

I'm posting this here as I wasn't seeing many others reporting this issue nor resolution and hope it saves someone else a lot of headache in solving it.

We setup AWS Inspector for our four AWS accounts to check for scan results once a day and we were seeing odd Rate Limit Exceeded Errors and only a limited number of results were getting pulled in.

2021-03-17 14:17:31,524 level=ERROR pid=32092 tid=Thread-6 logger=splunk_ta_aws.modinputs.inspector.aws_inspector_data_loader pos=aws_inspector_data_loader.py:__call__:307 | | message="Failed to collect inspector findings for region=us-east-1, datainput=Corp - Inspector, error=Traceback (most recent call last):
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 302, in __call__
self._do_indexing()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 284, in _do_indexing
AWSInspectorFindingsDataLoader(self._config, self._cli, account_id).run()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 176, in run
self._schedule()
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 191, in _schedule
arns = self._list_findings_by_time_window(begin, end)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py", line 249, in _list_findings_by_time_window
response = self._cli.list_findings(**params)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/data/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/client.py", line 586, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListFindings operation (reached max retries: 4): Rate exceeded
"

At once a day, it seems like we should not be hitting this ThrottlingException.

Labels (2)
Tags (2)
0 Karma
1 Solution

ejharts2015
Communicator

This issue is resolved by increasing the max number of results returned per API request. By default (AWS Documentation) the ListFindings maxResults returns 10 results per call. Increasing this small value to a higher one resolved our issue.

Python Script to modify:

 

/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py

 

And the function to modify:

 

 def _list_findings_by_time_window(self, begin, end):
    # boto3 do not accept unix timestamp on windows
    # cast to datetime by hand
    begin = datetime.datetime.utcfromtimestamp(begin)
    end = datetime.datetime.utcfromtimestamp(end)
    params = {
        "filter": {
            'creationTimeRange': {
                'beginDate': begin,
                'endDate': end
            }
        },
        "maxResults": 500
    }

 

Resulted in far less API calls and no further rate-limiting. I've put in an enhancement request for this function's defaults to be modified. Hope this helps!

API calls before/after change.API calls before/after change.

 

View solution in original post

0 Karma

ejharts2015
Communicator

This issue is resolved by increasing the max number of results returned per API request. By default (AWS Documentation) the ListFindings maxResults returns 10 results per call. Increasing this small value to a higher one resolved our issue.

Python Script to modify:

 

/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/inspector/aws_inspector_data_loader.py

 

And the function to modify:

 

 def _list_findings_by_time_window(self, begin, end):
    # boto3 do not accept unix timestamp on windows
    # cast to datetime by hand
    begin = datetime.datetime.utcfromtimestamp(begin)
    end = datetime.datetime.utcfromtimestamp(end)
    params = {
        "filter": {
            'creationTimeRange': {
                'beginDate': begin,
                'endDate': end
            }
        },
        "maxResults": 500
    }

 

Resulted in far less API calls and no further rate-limiting. I've put in an enhancement request for this function's defaults to be modified. Hope this helps!

API calls before/after change.API calls before/after change.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...