The question is about the configuration of max_cache_size for the smart store.
* Specifies the maximum space, in megabytes, per partition, that the cache can occupy on disk. If this value is exceeded, the cache manager starts evicting buckets.
* A value of 0 means this feature is not used, and has no maximum size.
* Default: 0
What is the point of setting this to anything but 0?
The max_cache_size puts a cap on the the amount of storage in a single volume consumed by SmartStore cache.
This allows customers to split the warm/cold storage consumption between SmartStore and non-SmartStore indexes on the same volume.
Since there are no caps placed on non-smartStore indexes (module minFreeSpace), setting max_cache_size to non-zero can result in non-SmartStore indexes squeezing out space available for SmartStore cache.
One usecase for max_cache_size in addition to migration usecase is for customers to reduce their local storage consumption in staged manner. By limiting storage available to the cache relative to the available space, customers can identify an optimal cache size before reducing the indexer count.
So max_cache_size means the total size taken by buckets downloaded from an S3 source?
This is not really clear in the documentation as it just says "the disk partition that contains the cache".
So max_cache_size shouldn't be too low to let Splunk download and keep the buckets locally for a while, right?
Yes that is right, you will at a very minimum want as least enough space in the cache to hold the buckets currently being searched. Ideally you'll want enough space to hold all of the buckets that are searched regularly. It might be worth looking at configuring hotlist_recency_seconds & hotlist_bloom_filter_recency_hours to control how long the cache manager will attempt to keep recent buckets in the cache, depending on your search patterns. These can be set at an index level, to prioritize certain indexes in the cahce, under each indexes stanza in indexes.conf or under the cachemanager stanza in server.conf for a global configuration. There are some specific cache sizing calculations here: https://answers.splunk.com/answers/766013/smartstore-sizing.html.
The Indexing -> SmartStore -> SmartStore Cache Performance: Deployment dashboard has a panel at the bottom to show if any cache thrashing is occurring and on which indexes, it might be worth reviewing this to make sure you don't have buckets being evicted then downloaded again shortly.
I have a query. When we don't place any cache manager setting in smartstore enabled server's server.conf file, will it be default =unlimited cache? what is the cache location, is it splunk_db only?