Getting Data In

How to troubleshoot or validate smart storage configuration

Explorer

Hi  ,

In our current Splunk infrastructure , indexes are enabled with smart store and indexers are clustered. Now our local storage is almost 80% full. 

When further validating ,i noticed that a particular index which is enabled with smart store stores the entire warm buckets in the indexers(local store). But according to my understanding only a partial of warm buckets will have a local copy and others hast to evicted right?

 

Could some one please help with the troubleshooting steps to ensure whether the smart store is properly configured

 

TIA

Labels (1)
Tags (1)
0 Karma

Explorer

Thank you @scelikok  . That clarified my question. Can we change the default 10Gb to higher? is it recommended . If so where this change can be done?

 

Thanks

0 Karma

Builder

Hi @spl_unker,

You're welcome. In forgot to mention, SmartStore recommendation is to keep maxDataSize as auto which is 700MB. The reason for this is make downloads from S3 to local cache faster. This will also help download less data. 

That is why better to use as auto.

If this reply helps you an upvote is appreciated.

0 Karma

Builder

Hi @spl_unker,

SmartStore uses local storage volume for hot buckets and as a cache for warms. Although it depends on your settings, having still %20 free space explains why entire warm buckets are in local volume. Eviction will start when your indexers free space gets lower than ('minFreeSpace' + 'eviction_padding') which is default 10 GB.

SmartStore will copy all warm buckets to remote storage immediately when they created. Until your local volume has free space more than ('minFreeSpace' + 'eviction_padding') there will be no eviction. In case of local storage reaches to maximum size, oldest warm buckets start to be evicted.   

For troubleshooting you can use below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/TroubleshootSmartStore 

If this reply helps you an upvote is appreciated.

Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!