How do I remove IOCs from a KV store?

[PT088]

When we first got Splunk ES, one of my colleagues decided to try adding in IOCs from the Mandiant APT1 report. These were added to the path:
/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/default/data/threat_intel/Appendix_G_IOCs_No_OpenIOC.xml

The problem is that these are constantly producing false positives from the ES “Threat Activity Detected” correlation search, matching on simple common file names like “setup.exe”.

I have removed the original .xml file, but the IOCs have obviously been ingested into the KV store. Is there any way that we can clear them out without affecting other IOCs that we want to keep?

I understand that there is a way to “age out” IOCs with expiry times. I think this is one of the lookup options, but I can’t for the life of me find that setting again or the documentation that led me to it.

Any advice appreciated.

[jwelch_splunk · Accepted solution]

So first you must understand how it works:

If you have data in the pickup dir you have to get rid of it. ( Which you say you did)

Second the data has to be removed from the kvstore, there are 9 collections that data is parsed into, you can look at the all_threat_intel macro and it should define them all.

You could do something like this:

|inputlookup ip_intel |stats count by threat_key

Once you figure out what your threat key is do something like:

|inputlookup ip_intel |search threat_key!="mythreat_key"

 This should return all results except for the ones you want to get rid of. After you verify that add this to the end:

|inputlookup ip_intel |search threat_key!="mythreat_key" |outputlookup ip_intel

 (This will overwrite the collection with everything EXCEPT the ones you want to remove.

Now that the kvstore no longer has the data, ES runs a series of "Lookup Gen" searches that will suck the data from the kvstore collections, and update CSV files in DA-ESS-ThreatIntelligence/lookups/threatintel_by_blahblah..
Now that the CSV's are current E.G. no longer contain your old threat_key...

When the series of "Threat Gen" searches run find your data and enrich it with the lookups on the threatintel_by_blahblah.csv files you will no longer get hits on the old collections that you removed.

You may also want to |delete the old junk in your index=notable / index=threat_activity pertaining to the stuff you just removed, or you can just wait a few days and they should slowly start disappearing from the dashboards.

You can also enable "Retention / Aging" . To do this you go the go to Intelligence Downloads and pick the one you want and in there you add a entry in the max age... But for this to be honored you also have to enable the threat retention searches. Here is the doc for this part:

https://docs.splunk.com/Documentation/ES/5.1.0/Admin/Changethreatintel

The cleaning of data from the kvstore can be a bit time consuming and danting, but can be done. You can also just clean all the collections and over the next several download cycles it will rebuild them with all the current data as well, if you don't care about all the old stuff in them.

E.G . 

|inputlookup ip_intel |search threat_key=NOTHING |outputlookup ip_intel

You just searched for somethign you won't find and re-write nothing back over the existing data so you blanked it out. Now the next time the parser runs it will update it again with the currrent / future downloads of data.

[riqbal47010]

Sorry still my problem not solved.

Still the "threat activity detected" is triggering against custom uploaded csv file.

However when I give below command and there is no data against that feed.
| inputlookup ip_intel | search threat_key="abc"

and under threat activity dashboard I can see that the custom csv upload threat feed is still ative against
most active threat sources

[PT088]

This is excellent. Thanks for taking the time to explain this.

[pmelon]

This has helped me too just now - excellent answer.