Optiv Threat Intel: Why can't I pull Tor Exit Nodes or Zeus data?

[todd_miller]

I installed this app yesterday and it's pulling all data except that from the Tor Exit Nodes and the Zeus blacklist (and I think a few others). Here's what the troubleshooting logs say:

[*] Script Started at: 12-16-2015 12:42:36 GMT
[*] Script version: 2.02
URL: http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
Finished retrieving 705 IPs from SpamHaus.
Finished retrieving 23 IPs from Dshield.
Finished retrieving 364 IPs from Feodo.
URL: http://rules.emergingthreats.net/blockrules/compromised-ips.txt
Finished retrieving 1046 Emerging Threats Compromised IPs.
URL: http://www.binarydefense.com/banlist.txt
Finished retrieving 13322 IPs from Binary Defense.
URL: http://malc0de.com/bl/IP_Blacklist.txt
Finished retrieving 944 malc0de_IPs.
URL: https://reputation.alienvault.com/reputation.generic
Finished retrieving 135722 IPs from AlienVault.
URL: https://check.torproject.org/exit-addresses
Something happened! Error code 500
URL: https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
Page not found!
URL: https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist
Page not found!
URL: http://www.openbl.org/lists/base_1days.txt
Finished retrieving 128 IPs from Open Blocklist base 1 day.
URL: http://avant.it-mate.co.uk/dl/Tools/hpHosts/hosts.txt
Finished retrieving 168239 HP Hosts by MalwareBytes Domains.
URL: http://www.malwaredomainlist.com/hostslist/hosts.txt
Finished retrieving 766 Malware Domains.
URL: https://isc.sans.edu/feeds/suspiciousdomains_High.txt
Finished retrieving 4167 ISC SANS Suspicious Domains.
URL: https://openphish.com/feed.txt
Finished retrieving 3283 Open Phish URLs.
URL: http://data.phishtank.com/data/online-valid.csv
Finished retrieving 26237 Phish Tank URLs.
[*] Executing get alerts script.

There's not a whole lot of other details given, at least none that I can find.

[sherly_627]

I have the exact same error.

I'm not sure why I couldn't get through website with https
the URL already whitelisted and it shouldn't be the proxy blocking anymore.

Problem occurred even I commented out the below:

raw_threatlist = getUrl(urlList[4].strip('\n'),'true') 
  #if len(str(raw_threatlist)) > 3:
      #parseAlienVault(raw_threatlist)

Any other resolution on this?

[jeffriesa]

i am getting the same error:

URL: https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
Page not found!

If i use curl with the proxy settings i get results?

[jeffriesa]

I have checked our proxies and everything else inline and can see the sites being accessed.

This is what happens when we run the script manually:

[*] Script version: 3.00
URL: http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
Finished retrieving 818 IPs from SpamHaus.
Finished retrieving 23 IPs from Dshield.
Finished retrieving 716 IPs from Feodo.
URL: http://rules.emergingthreats.net/blockrules/compromised-ips.txt
Finished retrieving 1423 Emerging Threats Compromised IPs.
URL: http://www.binarydefense.com/banlist.txt
Finished retrieving 4298 IPs from Binary Defense.
URL: https://check.torproject.org/exit-addresses
Something happened! Error code 500
URL: https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
Page not found!
Traceback (most recent call last):
File "/apps/splunk/etc/apps/optiv_threat_intel/bin/optiv_threat_lists.py", line 882, in
main()
File "/apps/splunk/etc/apps/optiv_threat_intel/bin/optiv_threat_lists.py", line 816, in main
parseAlienVault(raw_threatlist)
File "/apps/splunk/etc/apps/optiv_threat_intel/bin/optiv_threat_lists.py", line 702, in parseAlienVault
AlienVaultIPs = urlResults.split('# Generic format')
AttributeError: 'int' object has no attribute 'split'

[derekarnold]

I answered this in the other thread but here goes for completeness:

Dev here. I am getting the same thing. AlienVault changed something with their feed. I am working on a new release to address some issues. In the meantime, try this:

Open optiv_threat_lists.py
Go to line 817
Comment out the line specified.

 raw_threatlist = getUrl(urlList[4].strip('\n'),'true') 
 #if len(str(raw_threatlist)) > 3:
     #parseAlienVault(raw_threatlist)

^^^^^Put a hash mark in front of parseAlienVault

Restart splunk. Just tested it and it fixed the issue for me.

I am not sure why ZeuS wasn't showing up for you when you posted this, it just worked for me a few minutes ago however. Perhaps the site briefly went down.

[jeffriesa]

after commenting out it got past that feed.

Then fail on the next https feed.

To me without restarting that is, that the issue is when there is a https feed. As soon as i comment out any of https feeds to works?

[jeffriesa]

Not related to this in a way.

But we have our SPlunk running under /apps/splunk. The python script fails as it looks for /opt/splunk and tries to write a log to this location. So you have to edit the python script to change the location of splunk.

The Error message you get when running the python script:
logfile_name: /opt/splunk/var/log/splunk/optiv_threat_lists_script11-23-2016-23-05-20.log
Traceback (most recent call last):
File "./optiv_threat_lists.py", line 129, in
lf = open(logfile_name,'w')
IOError: [Errno 2] No such file or directory: '/opt/splunk/var/log/splunk/optiv_threat_lists_script11-23-2016-23-05-20.log'

[derekarnold]

Since most of the lists are pulling, it's probably not a proxy issue. Let's see if it keeps giving the same error during the next cycle which should be in the next couple hours. If you want to run it manually you can restart splunk, or run the command:

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/optiv_threat_intel/bin/optiv_threat_lists.py

Some of the lists' web sites have been spotty lately, especially ZeuS.
I do have pretty recent data on all of the lists on my end - as of two hours ago:

[*] Script Started at: 12-16-2015 17:53:41 GMT
[*] Script version: 2.10
URL: http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
Finished retrieving 705 IPs from SpamHaus.
Finished retrieving 23 IPs from Dshield.
Finished retrieving 371 IPs from Feodo.
URL: http://rules.emergingthreats.net/blockrules/compromised-ips.txt
Finished retrieving 1051 Emerging Threats Compromised IPs.
URL: http://www.binarydefense.com/banlist.txt
Finished retrieving 13322 IPs from Binary Defense.
URL: http://malc0de.com/bl/IP_Blacklist.txt
Finished retrieving 941 malc0de_IPs.
URL: https://reputation.alienvault.com/reputation.generic
Finished retrieving 125660 IPs from AlienVault.
URL: https://check.torproject.org/exit-addresses
Finished retrieving 1122 TorExitNodes.
URL: https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
Finished retrieving 189 IPs from Zeus.
URL: https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist
Finished retrieving 17 IPs from Palevo.
URL: http://www.openbl.org/lists/base_1days.txt
Finished retrieving 123 IPs from Open Blocklist base 1 day.
URL: http://avant.it-mate.co.uk/dl/Tools/hpHosts/hosts.txt
Finished retrieving 168239 HP Hosts by MalwareBytes Domains.
URL: http://www.malwaredomainlist.com/hostslist/hosts.txt
Finished retrieving 766 Malware Domains.
URL: https://isc.sans.edu/feeds/suspiciousdomains_High.txt
Finished retrieving 4167 ISC SANS Suspicious Domains.
URL: https://openphish.com/feed.txt
Finished retrieving 3358 Open Phish URLs.
URL: http://data.phishtank.com/data/online-valid.csv
Finished retrieving 26219 Phish Tank URLs.
URL: http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt
Finished retrieving 255 Bambenek IPs.
URL: http://www.talosintel.com/feeds/ip-filter.blf
Finished retrieving 43902 Talos Intel IPs.
[*] Executing get alerts script.

[todd_miller]

Very bizarre Derek. Here are the results of me doing a manual pull right now:

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/optiv_threat_intel/bin/optiv_threat_lists.py
logfile_name: /opt/splunk/var/log/splunk/optiv_threat_lists_script12-16-2015-19-52-06.log
[*] Script Started at: 12-16-2015 19:52:06 GMT

[*] Script version: 2.02
URL: http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
Finished retrieving 705 IPs from SpamHaus.
Finished retrieving 23 IPs from Dshield.
Finished retrieving 371 IPs from Feodo.
URL: http://rules.emergingthreats.net/blockrules/compromised-ips.txt
Finished retrieving 1051 Emerging Threats Compromised IPs.
URL: http://www.binarydefense.com/banlist.txt
Finished retrieving 13322 IPs from Binary Defense.
URL: http://malc0de.com/bl/IP_Blacklist.txt
Finished retrieving 941 malc0de_IPs.
URL: https://reputation.alienvault.com/reputation.generic
Finished retrieving 125678 IPs from AlienVault.
URL: https://check.torproject.org/exit-addresses
Something happened! Error code 500
URL: https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
Page not found!
URL: https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist
Page not found!
URL: http://www.openbl.org/lists/base_1days.txt
Finished retrieving 114 IPs from Open Blocklist base 1 day.
URL: http://avant.it-mate.co.uk/dl/Tools/hpHosts/hosts.txt
Finished retrieving 168239 HP Hosts by MalwareBytes Domains.
URL: http://www.malwaredomainlist.com/hostslist/hosts.txt
Finished retrieving 766 Malware Domains.
URL: https://isc.sans.edu/feeds/suspiciousdomains_High.txt
Finished retrieving 4167 ISC SANS Suspicious Domains.
URL: https://openphish.com/feed.txt
Finished retrieving 3374 Open Phish URLs.
URL: http://data.phishtank.com/data/online-valid.csv
Finished retrieving 26262 Phish Tank URLs.

[derekarnold]

Error code 500 is an HTTP "internal server error"

"Page not found" is equivalent to an HTTP 404 error.

So if you do the curl command without the --proxy parameter, does it not work? Are there any significant delays to any of the downloads via curl? The python code is using the urllib2 library.

I'm trying to isolate if it's a timeout of some sort, or if I need to look at adding organizational proxy support in some manner.

[todd_miller]

Nothing works without the --proxy parameter since we're in a non-transparent proxied environment. I can only assume that I've gotten the rest to work in the app because the urllib2 library is using the proxy defined in the "splunk-launch.conf" file where I have both http and https proxies defined.

Delays via curl are insignificant and not out of the norm. I do, however, get this when trying to download the palevotracker:

504 Gateway Timeout
Gateway Timeout
Server error - server 198.105.244.11 is unreachable at this moment.  

Please retry the request or contact your adminstrator.

    # Palevo C&C IP Blocklist by abuse.ch
    107.150.36.226
    162.159.210.67
    162.159.211.67
    173.230.133.99
    185.31.24.23
    185.68.16.107
    189.236.206.143
    199.2.137.20
    199.2.137.25
    54.183.180.82
    67.198.207.34
    67.210.170.169
    76.74.255.138
    82.196.6.164
    91.208.194.18

Despite the error at the beginning of the message, it does seem to download. No similar errors on the other two in question.

[jkat54]

I'm able to browse https://check.torproject.org/exit-addresses in a browser from my location. Are you able to do so from your splunk server?

curl -k https://check.torproject.org/exit-addresses

Same for https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist

curl -k https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist

And same for https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist

curl -k https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist

My guess is you've got these sites blocked for some reason. I'd check you web proxy logs, firewall logs, etc.

[todd_miller]

All of them work if I add the "--proxy" argument to the curl command.

[todd_miller]

Just validated the proxy logs and they're all returning error code 200 for all of the above domains.

Seems that the connectivity exists thru the proxy -- which is validated thru the manual checks as well.