Getting Data In

How to troubleshoot or validate smart storage configuration

spl_unker
Explorer

Hi  ,

In our current Splunk infrastructure , indexes are enabled with smart store and indexers are clustered. Now our local storage is almost 80% full. 

When further validating ,i noticed that a particular index which is enabled with smart store stores the entire warm buckets in the indexers(local store). But according to my understanding only a partial of warm buckets will have a local copy and others hast to evicted right?

 

Could some one please help with the troubleshooting steps to ensure whether the smart store is properly configured

 

TIA

Labels (1)
Tags (1)
0 Karma
1 Solution

scelikok
Champion

Hi @spl_unker,

SmartStore uses local storage volume for hot buckets and as a cache for warms. Although it depends on your settings, having still %20 free space explains why entire warm buckets are in local volume. Eviction will start when your indexers free space gets lower than ('minFreeSpace' + 'eviction_padding') which is default 10 GB.

SmartStore will copy all warm buckets to remote storage immediately when they created. Until your local volume has free space more than ('minFreeSpace' + 'eviction_padding') there will be no eviction. In case of local storage reaches to maximum size, oldest warm buckets start to be evicted.   

For troubleshooting you can use below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/TroubleshootSmartStore 

If this reply helps you an upvote is appreciated.

If this reply helps you an upvote is appreciated.

View solution in original post

scelikok
Champion

Hi @spl_unker,

You should not change server.conf on every indexer manually. You should send this server.conf to indexers via Cluster Master. There is no need maintenance mode either. 

After you push the new server.conf to indexers minFreeSpace and eviction_padding values will be active and indexers will make necessary processes.

If this reply helps you an upvote is appreciated.
0 Karma

spl_unker
Explorer

Thank you @scelikok  . That clarified my question. Can we change the default 10Gb to higher? is it recommended . If so where this change can be done?

 

Thanks

0 Karma

scelikok
Champion

Hi @spl_unker,

You're welcome. In forgot to mention, SmartStore recommendation is to keep maxDataSize as auto which is 700MB. The reason for this is make downloads from S3 to local cache faster. This will also help download less data. 

That is why better to use as auto.

If this reply helps you an upvote is appreciated.

If this reply helps you an upvote is appreciated.
0 Karma

scelikok
Champion

Hi @spl_unker,

SmartStore uses local storage volume for hot buckets and as a cache for warms. Although it depends on your settings, having still %20 free space explains why entire warm buckets are in local volume. Eviction will start when your indexers free space gets lower than ('minFreeSpace' + 'eviction_padding') which is default 10 GB.

SmartStore will copy all warm buckets to remote storage immediately when they created. Until your local volume has free space more than ('minFreeSpace' + 'eviction_padding') there will be no eviction. In case of local storage reaches to maximum size, oldest warm buckets start to be evicted.   

For troubleshooting you can use below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/TroubleshootSmartStore 

If this reply helps you an upvote is appreciated.

If this reply helps you an upvote is appreciated.

View solution in original post

spl_unker
Explorer

Hi @scelikok ,

I have decided to increase the  min free space and eviction padding . Before doing  this  , i  want to clarify on the steps .  It will be great if you could  validate my steps. I have 3 indexers in cluster with one Cluster master.

 

Step 1: I'm going to put indexers on maintenance mode 

Step 2 : Login into each the server.conf  and update the following settings: 

[diskusage]

minFreeSpace = “10%”

 

[cachemanager]

eviction_padding = 51200

 

Step 3 : Enable maintenance mode

 

Please validate the above steps and let me know if i missed anything.

 

Additional question, if my disk is already 95% full , is it safe to update now with minfreespace as 10%? I hope it will evict the buckets immediately to maintain 10 % free space and will not cause any other issue?

 

Thanks in advance

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.