topic Re: Smartstore, how to set max data size per index in Splunk Enterprise

Smartstore, how to set max data size per index

mufthmu — Tue, 27 Oct 2020 22:50:20 GMT

Hi fellow Splunkers,

As parameter maxTotalDataSizeMB is only available for non-Smartstore indexes, what parameter in Smartstore index that replace this parameter? I want to start evicting bucket when certain size is reached. I could only find hotlist_recency_secs which evict data by age, not size (per index).

Thanks!

Reference: https://docs.splunk.com/Documentation/Splunk/8.1.0/Admin/Indexesconf

Re: Smartstore, how to set max data size per index

richgalloway — Wed, 28 Oct 2020 00:17:11 GMT

Check out maxGlobalDataSizeMB and maxGlobalRawDataSizeMB.

The hotlist_recency_secs setting partially controls the S2 cache. That's different from controlling the data of the S2 database itself. which is what the other two settings above do.

Re: Smartstore, how to set max data size per index

mufthmu — Wed, 28 Oct 2020 15:20:07 GMT

Thanks @richgalloway

But I read the following for maxGlobalDataSizeMB:

* This setting includes the sum of the size of all buckets that reside
  on remote storage, along with any buckets that have recently rolled
  from hot to warm on a peer node and are awaiting upload to remote storage.

This parameter includes all warm buckets in local storage AND remote storage. Meaning that if I set this parameter to 200GB, I can only have data for maximum 200GB in BOTH local and remote storage (correct me if I'm wrong) for that index. I'm more interested in the parameter that only includes local storage, so that when that value is reached, the cache manager will begin evicting buckets to remote storage.

Re: Smartstore, how to set max data size per index

richgalloway — Wed, 28 Oct 2020 15:29:32 GMT

Recall that warm buckets do not exist locally. If a warm bucket is found on local storage it is because it is waiting to be transferred to remote storage. Local buckets are counted because space for them needs to be accounted for on the remote store.

To set the size of the SmartStore cache, use max_cache_size in server.conf. See https://docs.splunk.com/Documentation/Splunk/8.0.5/Indexer/ConfigureSmartStorecachemanager#Initiate_eviction_based_on_occupancy_of_the_cache.27s_disk_partition

Re: Smartstore, how to set max data size per index

mufthmu — Thu, 29 Oct 2020 16:16:43 GMT

@richgalloway I see.

So there is no parameter for Smartstore that limit data size locally by index, am I correct?

maxGlobalDataSizeMB = includes remote storage. (I only want to evict bucket in my S3 by age, not size)

max_cache_size = this is cache manager as a whole, not per index.

PS: It will be perfect for my situation if maxGlobalDataSizeMB does not include remote storage.

Re: Smartstore, how to set max data size per index

richgalloway — Thu, 29 Oct 2020 16:54:02 GMT

Correct, there are no per-index settings for SmartStore. S2 caches whatever indexes it needs to meet search requirements.

There are per-index size limit settings, but they do not apply when using SmartStore.