How do I get data from a particular website using the Getwatchlist Add-on for Splunk Enterprise?

[dmenon84]

Hi,

I am trying to load the data from website - https://www.malwaredomainlist.com/update.php into Splunk using getwatchlist. I tried the following, but this doesn't seem to work.

| getwatchlist https://www.malwaredomainlist.com/update.php categoryCol=3 ignoreFirstLine=true isbad=true | outputlookup malware_domainsdm.csv

Thanks in advance !

[dshpritz · Accepted solution]

Getwatchlist is meant for parsing delimited files. The link you posted is an HTML file, and as such getwatchlist is not going to parse it correctly. You may want to try this instead:

| getwatchlist malwaredomains | outputlookup malware_domainsdm.csv

As that particular list comes preconfigured in getwatchlist. If you only wanted something more similar to the updates page that you posted, you could try http://www.malwaredomainlist.com/updatescsv.php which produces a delimited version of that page. That might look like this:

| getwatchlist delimiter="," relevantFieldCol=2 relevantFieldName=domain isbad=true | outputlookup malware_domainsdm.csv

HTH,

Dave

[dmenon]

Wow that worked, thanks for your help! This is what I have now:

| getwatchlist http://www.malwaredomainlist.com/updatescsv.php delimiter="," relevantFieldCol=3 relevantFieldName=malware_IP isbad=true | outputlookup malware_IPs.csv

I ran this search for last 15 mins and saved it as a report, is that the correct way to do it if I want the watchlist to be periodically update ?
Also now that I have these IPs (malware_IP) I would like to compare them to the dest_ip field on my firewall.
Thanks in advance !

[dshpritz]

Getwatchlist doesn't keep track of the file. That is, if you get the updates file, you will only ever have the latest updates in your lookup. You may want to use this link instead: http://www.malwaredomainlist.com/mdlcsv.php

You can use your search as a saved search (getwatchlist doesn't care about the timeframe), which will then update your lookup on a regular basis.

Here is a search that would use the list contents to find events with the malware domains match:

index=firewall [| inputlookup malware_IPs.csv | fields malware_IP | rename malware_IP AS dest_ip]

If you run into subsearch limits (if the list of IPs is longer than 10k), then your search gets a little heavier:

index=firewall | lookup malware_IPs  dest_ip AS malware_IP | search isbad=true

[dmenon84]

Awesome ! It worked like I wanted it to, Thanks a lot !

[dmenon84]

Hi, One last question, why do you think http://www.malwaredomainlist.com/updatescsv.php will not work , http://www.malwaredomainlist.com/mdlcsv.php has domains and url from 2009 and all and is giving me lot of false positives ?

[dshpritz]

You certainly could use just the updated list, you just need to be aware of how often it is updated vs. how often you are updating it. That is, if it is updated every 3 hours, and you are only updating the local version every 6 hours, then you would be missing an update, and potentially events that you would be interested in.