All Apps and Add-ons

Shodan App, Proxy and Query Rate Limit

Path Finder

I've installed Hurricane Labs App for Shodan v2.0.1 on Splunk Enterprise v7.2.4 and I've found some issues in App usage and functionality.

- First Issue:
I need to use a Proxy for exit on Internet, and when I set Proxy globally on OS, I can contact all sites, but Shodan App cannot contact Shodan API through Proxy, App contact Shodan API directly (I've done some tcpdumps to hit the problem).

After some debugging, I've found the issue and modified line 173 on $SPLUNK_HOME/etc/apps/Hurricane_Labs_App_for_Shodan/bin/shodan/ file that reports

def __init__(self, key, proxies=None):
    """Initializes the API object.

    :param key: The Shodan API key.
    :type key: str
    :param proxies: A proxies array for the requests library, e.g. {'https': 'your proxy'}
    :type proxies: dict

now I can contact Shodan API through Proxy.
I suggest to make a modification on next App version, and add option to allow the user to modify Proxy Settings through App Web Interface on Splunk.

- Second Issue:
I need to add several subnets starting from /24 ending to /29, and after adding almost 20 subnets on "Configure Subnets" section, I receive a message that indicate the App cannot sync with Shodan:

No IPs to use. Add an IP above.

So I try to execute manual command to force list refresh:

| getshodan [|inputlookup shodan_my_subnets | stats values(ipAddress) AS ips | eval netlist=mvjoin(ips, ",")  | table netlist] | outputlookup shodan_output

and after some seconds it answer with Request rate limit reached:

APIError at "$SPLUNK_HOME/etc/apps/Hurricane_Labs_App_for_Shodan/bin/shodan/", line 255 : Request rate limit reached (1 request/ second). Please wait a second before trying again and slow down your API calls.

that's a problem, because App don't consider Shodan API Request rate limit, and this is a big problem.
Also I suggest to modify next App version with Shodan API Requests rate limit.

Any suggestion for quick resolve with a WA?

0 Karma

Path Finder

@morganfw thanks for reaching out. In regards to the second issue: as a potential quick fix you could modify on line 109 and add a time.sleep(1) to pause for 1 second before resuming with the next request.

So, it would look like:
for net in self.netlist:
query = "net:%s" % net
results +=['matches']

Let us know if that fixes the second issue for you.

Then we can work on implementing a permanent fix for the next release, as well as addressing the first issue.

0 Karma

Path Finder

The WA works like a charm and fixes the second issue.
I also suggest you to add some detailed documentation about App, to better understand all the functionalities.

Thank you for quick WA and for this awesome App.

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...