Splunk Enterprise Security

Local lookup CSV to Threat Intel KV timestamp - age out indicators

Path Finder

Has anyone tackled IOC expiry / timestamp issues between a local lookup and the Splunk ES Threat Intel KV store ?

I currently have a scheduled process that populates a local lookup with a list of indicators (IP / url / domain etc).

This local lookup is added as a threat intel download which collects the indicators and stores them within the Splunk ES Threat Intel KV stores (ip_intel / http_intel etc). This process works well, deduplication with existing indicators etc works, and I can see the indicators become available in the Threat Artifacts dashboard, and see matches from the Threat Gen searches etc etc.

Unfortunately every time the Threat Intel scripts collect the indicators from the local lookup and store them in the Threat Intel KV - they add a timestamp when the process runs. This means that each time it runs, its effectively a "new" indicator. I have set a maximum age of -7d on this list however this will never match as the timestamp is refreshed each time.

Populating the local lookup, I can only use the few fields available (e.g. - description,ip,weight) - it doesnt seem to support the addition of a timestamp which could be carried across to the KV store.

The macros responsible for cleaning up expired intel from the lists ("filter_threatintel_collection(1)" and "filter_by_threat_age") use the timestamp in the KV - which means the indicators will never match as they are updated every hour, and a new timestamp is added.

New Member

Did you ever find a solution to this? I'm currently dealing with the same exact scenario.

0 Karma

Path Finder

no, unfortunately I havent found a fully working solution yet. Incrementally updating the lookup table might work as the timestamps in the KV shouldnt change, but pulling the entire list each day refreshes the timestamps when the KV is populated - hence it will never seem to reach the 7 day age / expiry time.

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...