All Apps and Add-ons

How do I get data from a particular website using the Getwatchlist Add-on for Splunk Enterprise?

dmenon84
Path Finder

Hi,

I am trying to load the data from website - https://www.malwaredomainlist.com/update.php into Splunk using getwatchlist. I tried the following, but this doesn't seem to work.

| getwatchlist https://www.malwaredomainlist.com/update.php categoryCol=3 ignoreFirstLine=true isbad=true | outputlookup malware_domainsdm.csv

Thanks in advance !

0 Karma
1 Solution

dshpritz
SplunkTrust
SplunkTrust

Getwatchlist is meant for parsing delimited files. The link you posted is an HTML file, and as such getwatchlist is not going to parse it correctly. You may want to try this instead:

| getwatchlist malwaredomains | outputlookup malware_domainsdm.csv

As that particular list comes preconfigured in getwatchlist. If you only wanted something more similar to the updates page that you posted, you could try http://www.malwaredomainlist.com/updatescsv.php which produces a delimited version of that page. That might look like this:

| getwatchlist delimiter="," relevantFieldCol=2 relevantFieldName=domain isbad=true | outputlookup malware_domainsdm.csv

HTH,

Dave

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

Getwatchlist is meant for parsing delimited files. The link you posted is an HTML file, and as such getwatchlist is not going to parse it correctly. You may want to try this instead:

| getwatchlist malwaredomains | outputlookup malware_domainsdm.csv

As that particular list comes preconfigured in getwatchlist. If you only wanted something more similar to the updates page that you posted, you could try http://www.malwaredomainlist.com/updatescsv.php which produces a delimited version of that page. That might look like this:

| getwatchlist delimiter="," relevantFieldCol=2 relevantFieldName=domain isbad=true | outputlookup malware_domainsdm.csv

HTH,

Dave

View solution in original post

dmenon
Explorer

Wow that worked, thanks for your help! This is what I have now:

| getwatchlist http://www.malwaredomainlist.com/updatescsv.php delimiter="," relevantFieldCol=3 relevantFieldName=malware_IP isbad=true | outputlookup malware_IPs.csv

I ran this search for last 15 mins and saved it as a report, is that the correct way to do it if I want the watchlist to be periodically update ?
Also now that I have these IPs (malware_IP) I would like to compare them to the dest_ip field on my firewall.
Thanks in advance !

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Getwatchlist doesn't keep track of the file. That is, if you get the updates file, you will only ever have the latest updates in your lookup. You may want to use this link instead: http://www.malwaredomainlist.com/mdlcsv.php

You can use your search as a saved search (getwatchlist doesn't care about the timeframe), which will then update your lookup on a regular basis.

Here is a search that would use the list contents to find events with the malware domains match:

index=firewall [| inputlookup malware_IPs.csv | fields malware_IP | rename malware_IP AS dest_ip]

If you run into subsearch limits (if the list of IPs is longer than 10k), then your search gets a little heavier:

index=firewall | lookup malware_IPs  dest_ip AS malware_IP | search isbad=true
0 Karma

dmenon84
Path Finder

Awesome ! It worked like I wanted it to, Thanks a lot !

0 Karma

dmenon84
Path Finder

Hi, One last question, why do you think http://www.malwaredomainlist.com/updatescsv.php will not work , http://www.malwaredomainlist.com/mdlcsv.php has domains and url from 2009 and all and is giving me lot of false positives ?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

You certainly could use just the updated list, you just need to be aware of how often it is updated vs. how often you are updating it. That is, if it is updated every 3 hours, and you are only updating the local version every 6 hours, then you would be missing an update, and potentially events that you would be interested in.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!