topic Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data? in Splunk Search

Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

santorof — Fri, 18 Sep 2015 17:59:50 GMT

I just completed importing a CSV file as a threat intelligence lookup list. I followed Splunk documentation (6.2) and one of the details for threats by IP address is that IPs and descriptions are required. The file I was given only had IP addresses, so I changed under parsing options to only look at the first column for IPs with nothing defining descriptions. My question is, how can I test that this threat intelligence document is being parsed against my data and would files only with IPs still function without a description field?

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

woodcock — Fri, 18 Sep 2015 19:26:13 GMT

Have you seen the GetWatchList app?

https://splunkbase.splunk.com/app/635/

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

santorof — Mon, 21 Sep 2015 20:00:37 GMT

I looked at it but was not looking to add any apps that weren't built by splunk or splunk supported. Would I be able to install this app on my local machine and extract the information I would need to compare my data against specific threat lists?

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

woodcock — Tue, 22 Sep 2015 07:45:18 GMT

Yes, that is the whole point; you give it the URL where it can get the threatlist and it does the rest.

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

santorof — Fri, 25 Sep 2015 12:19:38 GMT

Sorry I did not make it clear. I have a local copy of splunk with no data I use as my test environment. The real data and infrastructure is what I don't want to install the app on. If it was xml or dashboard logic I could extract it from my local version but because its a command I dont believe it will help me with the real data.

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

sheamus69 — Fri, 25 Sep 2015 12:53:47 GMT

This is how I would do it (assuming that an existing threat intel field exists, called the-threat-intel-list.csv):

Upload "Update.csv" into search app
Change permissions on Update to App level
Once App viewable, you will be able to delete Update once done
Craft Splunk search to load in previous threat list, then append on the Update file (assuming updates go on the end) - this will NOT make any permanent changes at this point, merely display the output : | inputlookup the-threat-list.csv | inputlookup append=t Update.csv 4a. Dedup on certain columns, if desired (quoted only if special characters, such as periods or spaces, in field names)
Inspect search results to ensure columns match up properly (generally not a problem, in Splunk, beware if updating a CSV from python)
When confident list is as desired, write out to file: | inputlookup the-threat-list.csv | inputlookup append=t Update.csv | dedup [field to dedup if desired] | outputlookup the-threat-list.csv
Inspect file was written as intended: | inputlookup the-threat-list.csv

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

santorof — Fri, 25 Sep 2015 19:34:18 GMT

I did the above but have more questions. The threat list(the-threat-list) does show all the IP's I want to filter out by. So my logic is comparing these ip's in the IP field to a field called IP in my sourcetype firewall for example(checking for hits manually). Would I use the diff command to compare the IP field from the-threat-list to my sourcetype firewall field=IP for any matches? Just not sure how to go about it. I also tried to do the threatlookup command with files that are being pulled down from a url but got no results.

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

rashid47010 — Tue, 23 Apr 2019 17:05:56 GMT

HI Ninjas,

I upload custom threat intelligence. file_name, description,url
the threat activity detected notable is triggering successfully against custom threat intelligence that I have upload previously.
Now I want to remove those threat intelligence feeds. that threat activity detected should not trigger against that custom feeds.

Re: Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

woodcock — Tue, 23 Apr 2019 19:43:37 GMT

This anti-open-source bias is silly and severely limiting, but it is your system.