Why isn't Splunk Cloud 90-day searchable retention configuration deleting old data?

[untieshoe]

Hello,

I have Splunk Cloud 90-day searchable retention set for all indexes by default.

I created a new index with only 2-day retention (intentional). The index filled with data as intended. But data older than 2 days did not get deleted. The index continues to grow regardless of the "Searchable Retention = 2 days" configuration. What's up with that? This is a new Splunk Cloud environment, although at v7.2.10.1. From the 'Data Quality' Monitoring Console, I see the data is currently in 6 buckets and I have 1,730,000 events in the index. 1.2 GB of data.

Any advice on why this is happening would be appreciated.

[imsidrai]

Hi , were you able to fix the issue ?

[untieshoe]

It turned out to be a software bug. It does work now (sort of). I set the index size to 0 (no limit) and retention to 2 days. I can actually search 3 days, but that's close enough for my needs...

[isoutamo]

Hi

I think that this is due to Splunk's feature that it can manage only the whole buckets. This means that it can remove/delete the bucket when all data inside it is older than your retention time. Usually that leads to situation when you have some searchable events which are much older that what you have configured into indexes. Also all indexers have usually 3 open hot buckets with some default time (90days) before it rolls to then warm (or e.g. manually with REST or restart splunkd). As all Splunk Cloud instances has at least 3 indexers (usually more) this lead quite a many open hot buckets which contains older than X days data.

Here is splunk ingest flow https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor... where you can see how data goes between buckets. Here is old conf presentation https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-... which cover this more detail level. It's little bit old, but mainly valid. In Splunk Cloud as all warm and cold data are in SmartStore there are some difference in detail level, but I think that you can get the idea from that presentation?

r. Ismo