We've set up some Intelligence Downloads. These are downloading files from repository, on which they are upkept concerning retention (the available file is always up to date, so old entries get removed).
Since we'd like to have the same intelligence in Splunk that keeps up with it, we've set retention (Maximum age) on downloads to lowest possible -1d and interval is set at 1800. Issue seems to be that the downloads do not refresh the time, for example in ip_intel lookup, so the retention clears the still relevant IOCs, even when the files are successfully downloaded every 30 minutes. After being deleted these don't reappear on the next download either
Simply disable/enable on the downloads makes all of them work for one time, but after 24h most gets removed again as the time in collection doesn't refresh on every download.
Can't find any errors from anywhere and around 30% of the downloaded files seem to work a bit better (being added at least sometimes during the 24 hour period, but still not every 30 minutes) Settings and naming convention (no spaces) for all downloads is the same
Threat Intelligence Audit doesn't show any errors. Based on it the lists do get downloaded every 30 minutes, for example
status="threat list downloaded" file="/opt/splunk/var/lib/splunk/modinputs/threatlist/fqdn_critical.txt" bytes="1514"
What are some other places to look for errors? or is this somehow expected behavior, let's say if the downloaded file is exactly the same as previously it doesn't process it?
Expected behavior:
- Every 30 minutes every line in the downloaded file is refreshed to related intel lookup and to Threat Artifacts
Current behavior:
- Some of the threatlists get sometimes refreshed, most only work one time when disabling and re-enabling the download from Intelligence Downloads
