Shodan App, Proxy and Query Rate Limit

morganfw — Wed, 30 Sep 2020 00:53:50 GMT

Hello,
I've installed Hurricane Labs App for Shodan v2.0.1 on Splunk Enterprise v7.2.4 and I've found some issues in App usage and functionality.

- First Issue:
I need to use a Proxy for exit on Internet, and when I set Proxy globally on OS, I can contact all sites, but Shodan App cannot contact Shodan API through Proxy, App contact Shodan API directly (I've done some tcpdumps to hit the problem).

After some debugging, I've found the issue and modified line 173 on $SPLUNK_HOME/etc/apps/Hurricane_Labs_App_for_Shodan/bin/shodan/client.py file that reports

def __init__(self, key, proxies=None):
    """Initializes the API object.

    :param key: The Shodan API key.
    :type key: str
    :param proxies: A proxies array for the requests library, e.g. {'https': 'your proxy'}
    :type proxies: dict
    """

now I can contact Shodan API through Proxy.
I suggest to make a modification on next App version, and add option to allow the user to modify Proxy Settings through App Web Interface on Splunk.

- Second Issue:
I need to add several subnets starting from /24 ending to /29, and after adding almost 20 subnets on "Configure Subnets" section, I receive a message that indicate the App cannot sync with Shodan:

No IPs to use. Add an IP above.

So I try to execute manual command to force list refresh:

| getshodan [|inputlookup shodan_my_subnets | stats values(ipAddress) AS ips | eval netlist=mvjoin(ips, ",")  | table netlist] | outputlookup shodan_output

and after some seconds it answer with Request rate limit reached:

APIError at "$SPLUNK_HOME/etc/apps/Hurricane_Labs_App_for_Shodan/bin/shodan/client.py", line 255 : Request rate limit reached (1 request/ second). Please wait a second before trying again and slow down your API calls.

that's a problem, because App don't consider Shodan API Request rate limit, and this is a big problem.
Also I suggest to modify next App version with Shodan API Requests rate limit.

Any suggestion for quick resolve with a WA?
Regards

Re: Shodan App, Proxy and Query Rate Limit

hurricanelabs — Wed, 12 Jun 2019 14:01:44 GMT

@morganfw thanks for reaching out. In regards to the second issue: as a potential quick fix you could modify getshodan.py on line 109 and add a time.sleep(1) to pause for 1 second before resuming with the next request.

So, it would look like:
for net in self.netlist:
time.sleep(1)
query = "net:%s" % net
results += api.search(query)['matches']

Let us know if that fixes the second issue for you.

Then we can work on implementing a permanent fix for the next release, as well as addressing the first issue.

Re: Shodan App, Proxy and Query Rate Limit

morganfw — Wed, 12 Jun 2019 14:52:48 GMT

The WA works like a charm and fixes the second issue.
I also suggest you to add some detailed documentation about App, to better understand all the functionalities.

Thank you for quick WA and for this awesome App.

topic Re: Shodan App, Proxy and Query Rate Limit in All Apps and Add-ons

Shodan App, Proxy and Query Rate Limit

Re: Shodan App, Proxy and Query Rate Limit

Re: Shodan App, Proxy and Query Rate Limit