Smartstore:SmartStore cache is not respecting cache limits

rbal_splunk — Wed, 30 Sep 2020 04:22:54 GMT

Smartstore doesn't appear to be respecting our disk usage limits via the DiskUsage & CacheMan stanzas (minFreeSpace & max_cache_size respectively). Is there way to purge smartstore data that is using this excessive space? i've attached a diag from one of our peers & our cluster master.

Re: Smartstore:SmartStore cache is not respecting cache limits

rbal_splunk — Wed, 30 Sep 2020 04:22:56 GMT

Splunk has attribute max_cache_size that set the limit for cacahemanager

server.conf 
[cachemanager]
eviction_padding = 10% of disk Space on Partition (in bytes)
Refer: https://docs.splunk.com/Documentation/Splunk/7.1.4/Admin/Serverconf
max_cache_size=0

Given that max_cache_size is not accounting for hot data, it is likely that space usage is going to be above max_cache_size.o limit the disk usage for both hot data + cached data, you can use minFreeSpace to restrict total disk usage. However, you need to have reasonable total space so hot buckets are not overcrowding the cache.
Splunk component CacheManager when put in debug mode provides the stats for key attribute that contribute to the size of Cacahemanager.

02-05-2020 21:09:41.898 +0000 DEBUG CacheManager - The system has freebytes=75354976256 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=313491456 maxSize=314572800

"The system has freebytes=" > freeBytes
" with minfreebytes=" > minFreeBytes
" cachereserve=" > evictionReservedBytes
" totalpadding=" > minFreeBytes + evictionReservedBytes
" buckets_size=" > buckets_size
" maxSize=" > maxSize

Now here is some information on each of this attribute.

1) " maxSize=" << _maxSize;

Now this comes from

*max_cache_size = *
* Specifies the maximum space, in megabytes, per partition, that the cache can
occupy on disk. If this value is exceeded, the cache manager starts
evicting buckets.
* A value of 0 means this feature is not used, and has no maximum size.
* Default: 0

2) " buckets_size=" << buckets_size
Now this is the total size of the buckets calculated.

3) " totalpadding=" << minFreeBytes + evictionReservedBytes

we can see here that totalpadding=10611589120 = (minfreebytes+cachereserve) = (5242880000+5368709120)

From the below, we can see that buckets_size can get bigger than maxSize for a temporary period but eventually we will slash it down to maxSize(max_cache_size) OR lesser.

Here in this example, we have

As you can see how buckets_size is varying with maxSize wrt time. Once we have buckets_size growing above maxSize, the CacheManager will ensure that we evict something and get back within maxSize.
server.conf

[cachemanager]
max_cache_size = 300

topic Smartstore:SmartStore cache is not respecting cache limits in Knowledge Management

Smartstore:SmartStore cache is not respecting cache limits

Re: Smartstore:SmartStore cache is not respecting cache limits