Local lookup CSV to Threat Intel KV timestamp - age out indicators

[ahartge]

Has anyone tackled IOC expiry / timestamp issues between a local lookup and the Splunk ES Threat Intel KV store ?

I currently have a scheduled process that populates a local lookup with a list of indicators (IP / url / domain etc).

This local lookup is added as a threat intel download which collects the indicators and stores them within the Splunk ES Threat Intel KV stores (ip_intel / http_intel etc). This process works well, deduplication with existing indicators etc works, and I can see the indicators become available in the Threat Artifacts dashboard, and see matches from the Threat Gen searches etc etc.

Unfortunately every time the Threat Intel scripts collect the indicators from the local lookup and store them in the Threat Intel KV - they add a timestamp when the process runs. This means that each time it runs, its effectively a "new" indicator. I have set a maximum age of -7d on this list however this will never match as the timestamp is refreshed each time.

Populating the local lookup, I can only use the few fields available (e.g. - description,ip,weight) - it doesnt seem to support the addition of a timestamp which could be carried across to the KV store.
https://docs.splunk.com/Documentation/ES/5.2.2/Admin/Supportedthreatinteltypes

The macros responsible for cleaning up expired intel from the lists ("filter_threatintel_collection(1)" and "filter_by_threat_age") use the timestamp in the KV - which means the indicators will never match as they are updated every hour, and a new timestamp is added.

[jpolcari22]

Did you ever find a solution to this? I'm currently dealing with the same exact scenario.

[ahartge]

no, unfortunately I havent found a fully working solution yet. Incrementally updating the lookup table might work as the timestamps in the KV shouldnt change, but pulling the entire list each day refreshes the timestamps when the KV is populated - hence it will never seem to reach the 7 day age / expiry time.