How to troubleshoot or validate smart storage configuration

[spl_unker]

Hi  ,

In our current Splunk infrastructure , indexes are enabled with smart store and indexers are clustered. Now our local storage is almost 80% full. 

When further validating ,i noticed that a particular index which is enabled with smart store stores the entire warm buckets in the indexers(local store). But according to my understanding only a partial of warm buckets will have a local copy and others hast to evicted right?

Could some one please help with the troubleshooting steps to ensure whether the smart store is properly configured

TIA

[scelikok]

Hi @spl_unker,

You should not change server.conf on every indexer manually. You should send this server.conf to indexers via Cluster Master. There is no need maintenance mode either. 

After you push the new server.conf to indexers minFreeSpace and eviction_padding values will be active and indexers will make necessary processes.

[spl_unker]

Thank you @scelikok  . That clarified my question. Can we change the default 10Gb to higher? is it recommended . If so where this change can be done?

Thanks

[scelikok]

Hi @spl_unker,

You're welcome. In forgot to mention, SmartStore recommendation is to keep maxDataSize as auto which is 700MB. The reason for this is make downloads from S3 to local cache faster. This will also help download less data. 

That is why better to use as auto.

If this reply helps you an upvote is appreciated.

[scelikok · Accepted solution]

Hi @spl_unker,

SmartStore uses local storage volume for hot buckets and as a cache for warms. Although it depends on your settings, having still %20 free space explains why entire warm buckets are in local volume. Eviction will start when your indexers free space gets lower than ('minFreeSpace' + 'eviction_padding') which is default 10 GB.

SmartStore will copy all warm buckets to remote storage immediately when they created. Until your local volume has free space more than ('minFreeSpace' + 'eviction_padding') there will be no eviction. In case of local storage reaches to maximum size, oldest warm buckets start to be evicted.   

For troubleshooting you can use below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/TroubleshootSmartStore 

If this reply helps you an upvote is appreciated.

[spl_unker]

Hi @scelikok ,

I have decided to increase the  min free space and eviction padding . Before doing  this  , i  want to clarify on the steps .  It will be great if you could  validate my steps. I have 3 indexers in cluster with one Cluster master.

Step 1: I'm going to put indexers on maintenance mode 

Step 2 : Login into each the server.conf  and update the following settings: 

[diskusage]

minFreeSpace = “10%”

[cachemanager]

eviction_padding = 51200

Step 3 : Enable maintenance mode

Please validate the above steps and let me know if i missed anything.

Additional question, if my disk is already 95% full , is it safe to update now with minfreespace as 10%? I hope it will evict the buckets immediately to maintain 10 % free space and will not cause any other issue?

Thanks in advance