We are seeing this error:
2015-12-16 08:02:56,545 ERROR pid=42684 tid=MainThread file=protocols.py:run:226 | Caught HTTPError when querying http://data.phishtank.com/data/online-valid.csv.gz: code=509 exc=HTTP Error 509: Bandwidth Limit Exceeded
Splunk Enterprise Security App provides several pre-configured Threat Intelligence download sites in the OOB configuration that are avilable for you to enable and use.
These sites are operated / maintained by organizations outside of Splunk. These vendors have some limitations to their free offerings.
This error is one example of that: Phishtank limits access to this free service to 75 connections in a rolling 72 hour period.
At present Threat Intelligence downloads when enabled are done on the ES SH, and done every 12 hours. (configurable)
So one can infer that 1 ES SH would touch Phish Tank 2 times a day, but because we download the Threat Intelligence on each SH, if you are running an 5 node SHC for ES, this would grow 10 connections a day. Of course this excludes system restarts which can also trigger a download.
Customers that are PAT'ing/SNAT'ing their hosts leaving for the internet might have other systems in the Enterprise that also use these free services which would appear to Phishtank as all coming from the same system.
So it easy to see how this can become an issue.
Solution Possibilities:
1. Pay for a subscription service with these vendors and often times the connection limit will be removed.
2. Run a search on your Splunk Servers looking at your firewall data ( = and see if other hosts are also destined for those same destination addresses, if they are and you are hiding all hosts behind a firewall as mentioned earlier work with the other admins to tune down how often you are reaching out, as Phishtank would see you all to be the same source address.
3. Tune down the frequency on your SHC Nodes so you are not hitting the limits.
Okie
hi Okie @jwelch
if you register with Phishtank you can register an app and they give you a key (a long alphanumeric chain) to remove the limit. Is there any way in Splunk of configuring that key for the threatlist download?
Is it a URL modification? Is it a userid/password (auth) combo?
I would assume it would have to be tied to one of these. And both of those should be configurable.
If you want to send me what they sent you I can take a look at it.
okie at splunk / com