When does uploaded threat intelligence expire?

[npavlidis]

When a file is manually uploaded in Enterprise Security(ES), you can (and have to) define File Name, File to be uploaded, Weight, Threat Category, Threat Group, the option to Overwrite and the option to Sinkhole the file.
In our case I would not like to keep the file, just use the data for 24h so I will be selecting to Sinkhole it yet I am not sure how to configure data expiration. When a feed is configured in Threat Intelligence Downloads an expiration field is available, yet not the case here.

Thank you in advance.

N

[rlalwani_splunk · Accepted solution]

Select the sinkhole option that will delete the file once the intelligence manager consumes the data.
The Threat Intelligence manager will consume the useful information from the file and then stores it into KV store threat collections.
You can use Threat Intelligence API later and do CRUD operations on the same. This KV store based lookup contains the time when the threat intelligence manager consumed the data and you can use this time field for comparison as well.
Here is the link of the API: http://docs.splunk.com/Documentation/ES/5.0.0/API/ThreatIntelligenceAPIreference
Hope this helps you!!!!

[npavlidis]

I'll write a correlation search to use that field to clean the kv store then and run it on a defined schedule, sounds like a good plan, thanks!

[rlalwani_splunk]

You should not be cleaning the entire KV store lookup, as that lookup is defined by DA-ESS-ThreatIntelligence and other add-on's will be using the same, so you can just fetch the _key of the row that you want to delete and just hit the endpoint using that key.
You can delete multiple rows by a single call.

[npavlidis]

Yeah of course 😛 no need to reinvent the wheel there.. shame it's not part of ES by default though.. might have to request it as an enhancement