How to update a lookup periodically?

[CYBR_AH]

Hi All,

I'm wondering what would be the best way to download the latest CSV from http://cyberthreatalliance.org/cryptowall-dashboard.html

This site has information on the latest Cryptowall information (URL, IPs, Hashes, etc). I'd like to download the csv maybe once a month and update/replace the existing lookup. What would be the best way to do that? Also, would this cause any issues?

[woodcock]

You need the GetWatchList app:
https://splunkbase.splunk.com/app/635/

[yannK]

Once you know the location of the csv lookup on disk $SPLUNK_HOME/etc/apps/myapp/lookups/mylookup.csv
you can :

• create a script to replace the file. It can be done while splunk is running.
• or index the new csv file, and schedule a splunk search to
  
  • 1) return the results (in the correct field order)
  • 2) use the command "| outputlookup " to overwrite the existing lookup with the results.

see http://docs.splunk.com/Documentation/Splunk/6.3.2/SearchReference/Outputlookup

[renjith_nair · Accepted solution]

Your question is too general to answer for splunk. However, you can update your lookup files using a scheduler.

Run a scheduled script every month and download the details to lookup location of splunk , for eg: in linux use a script to run a curl command. It's recommended to take a back up of your previous file before updating it.

In case you want the data to be indexed instead of lookup, use splunk's input method with custom scripts.