[Smartstore] Can indexes spread across two file systems, can the max_cache_size setting be set per file system? Or is it global ?

[rbal_splunk]

Deployment has 30 indexers and 100 indexes
Each indexer has two 10TB filesystem ( /data1/indexes/... and /data2/indexes/..) . and both file systems have active indexes.
Half of the indexes use homepath of /data1/indexes/... and other have /data2/indexes/..)
How does a 5TB “max_cache_size” setting work?
Is it when either one of the filesystems reaches 5TB, is it when both file systems combined reach 5TB.
I would expect it to be both file systems combined reach 5TB.

[rbal_splunk · Accepted solution]

max_cache_size corresponds to a file system as opposed to the complete set of filesystems.
Hence the setting kicks in when either of the file systems reaches 5TB.

One downside with evicting buckets when "both file systems combined reach 5TB" is that
both of the file systems would not be used optimally as in buckets would get evicted when,
say both of them are at 2.5 TB thus ending up never using the other 5TB.

Hence max_cache_size is considered on a standalone basis

[gjanders]

Would you mind clarifying this answer, I'm not sure I understand the answer here.

Are you saying max_cache_size applies globally so that if either filesystem reaches 5TB in your example then eviction occurs?
Or that the combined filesystems reach 5TB then eviction starts occurring?

[rbal_splunk]

When the combined filesystems reach 5TB then eviction starts occurring?