Splunk Enterprise Security

Local lookup CSV to Threat Intel KV timestamp - age out indicators

Path Finder

Has anyone tackled IOC expiry / timestamp issues between a local lookup and the Splunk ES Threat Intel KV store ?

I currently have a scheduled process that populates a local lookup with a list of indicators (IP / url / domain etc).

This local lookup is added as a threat intel download which collects the indicators and stores them within the Splunk ES Threat Intel KV stores (ip_intel / http_intel etc). This process works well, deduplication with existing indicators etc works, and I can see the indicators become available in the Threat Artifacts dashboard, and see matches from the Threat Gen searches etc etc.

Unfortunately every time the Threat Intel scripts collect the indicators from the local lookup and store them in the Threat Intel KV - they add a timestamp when the process runs. This means that each time it runs, its effectively a "new" indicator. I have set a maximum age of -7d on this list however this will never match as the timestamp is refreshed each time.

Populating the local lookup, I can only use the few fields available (e.g. - description,ip,weight) - it doesnt seem to support the addition of a timestamp which could be carried across to the KV store.

The macros responsible for cleaning up expired intel from the lists ("filter_threatintel_collection(1)" and "filter_by_threat_age") use the timestamp in the KV - which means the indicators will never match as they are updated every hour, and a new timestamp is added.

New Member

Did you ever find a solution to this? I'm currently dealing with the same exact scenario.

0 Karma

Path Finder

no, unfortunately I havent found a fully working solution yet. Incrementally updating the lookup table might work as the timestamps in the KV shouldnt change, but pulling the entire list each day refreshes the timestamps when the KV is populated - hence it will never seem to reach the 7 day age / expiry time.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...