Losing data from URL-based Threat Intelligence Feed

[aingragunathan]

Hi All,

Looking for some help troubleshooting some odd behaviour around storing IOCs from a custom URL-based Threat Intelligence feed.
We have successfully set it up to a point where we can receive the IOCs (in 2hr intervals), store them and search with them.

But the IOCs seem to randomly disappear. One moment we may have 5000+ IOCs, the next we may have 0 or 2000 or 4000.
Our Threat Intelligence Management page states that the max size DA-ESS-ThreatIntelligence is 100MB and I haven't seen the threat_intel files pass 40MB

Any help troubleshooting this issue is appreciated!