I have an index with the hotlist_recency_secs set to 90 days (7776000 seconds). In testing I ran searches for the past 30 days. I noticed in the "Smartstore Cache Performance" dashboard that this 30 days search is triggering "misses". Meaning that this 30 days search is downloading buckets from S3. Why would a 30 days search on an index with hotlist_recency_secs set to 90 days need to download buckets for only a 30 day timeframe? I assume either I have something misconfigured, or more likely, I am not fully understanding the eviction process? the cachemanager, homePath setting, has 15tb per indexer so I don't believe its running out of space.
The hotlist_recency_secs setting does not prevent a bucket from being evicted from the cache. The goal is to keep the bucket in the cache for (at least) that long, but other demands on the cache may force an early eviction.
We can't tell is 15TB is enough cache space without knowing the ingestion rate. For example, at 1TB/day, it would only hold 15 days of data.