Why is the volume not restricting index size?

[rlaan]

/opt/splunk/etc/deployment-apps/indexer_config/local/indexes.conf

[volume:indexer_disk_size]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 530000

[network]
homePath = volume:indexer_disk_size/network/db
coldPath = volume:indexer_disk_size/network/colddb
thawedPath = $SPLUNK_DB/network/thaweddb
maxDataSize = auto_high_volume
maxHotIdleSecs = 86400
maxWarmDBCount = 7
frozenTimePeriodInSecs = 7776000
# 1 month = 2592000, 3 month = 7776000

I am experiencing that all of my indexes are able to individually go to a max of 500Gb which i believe is the default value. I am using Splunk 7.3.x.  Do i need to change any of my configuration settings, i suspect the volume referencing SPLUNK_DB as the path is causing issues although i have had difficulty.

I have many indexes in this volume that are collectively consuming 720GB of the attempted 530GB configuration (the 530 value was used to differentiate from the baseline 500GB).

Please help me understand what i have done wrong, the configurations are all done on the deployment server in indexes.conf and i am unable to see the configurations or deployment server defined index/volumes on any of my splunk monitoring consoles. 

Thanks.

[scelikok]

Hi @rlaan,

Yes, the same path can be defined on different volumes. Can you please share output of below query?

| rest /services/data/index-volumes
            | fields title, total_size, max_size, volume_path
            | eval total_size_gb = if(isnull(total_size), "-", round(total_size / 1024, 2))
            | eval max_size_gb = if(isnull(max_size) OR max_size = "infinite", "unlimited", round(max_size / 1024, 2))
            | eval disk_usage_gb = total_size_gb." / ".max_size_gb
            | fields title, disk_usage_gb, max_size_gb, volume_path
            | rename title as Volume, disk_usage_gb as "Volume Usage (GB)", max_size_gb as "Volume Capacity (GB)", volume_path as "Volume Path"

[rlaan]

Here is the output from requested query, i noticed the max size unlimited today in _introspection logs, although am still unsure how to change it, i thought the deployment app pushed to indexers would manage that with indexers.conf

Volume Volume Usage (GB) Volume Capacity (GB) Volume Path

_splunk_summaries

- / unlimited

unlimited

/opt/splunk/var/lib/splunk

My volume wasn't returned in your search, my reference to it being size infinite is from the below _introspection line.

{"datetime":"01-22-2021 13:32:17.668 -0700","log_level":"INFO","component":"Volumes","data":{"name":"indexer_disk_size","max_size":"infinite","volume_path":"/opt/splunk/var/lib/splunk"}}

[scelikok]

Hi @rlaan,

If the search does not show your volume, there is something wrong, also the monitoring console is not showing.

Are you using an indexer cluster? If yes these settings should be done on cluster master. I didn't ask at the first time assuming there is no cluster.

[rlaan]

We are setup with a following main servers
1- search head
1- deployment/license server
2- heavy forward
2- indexer (no replication) 

[scelikok]

Hi @rlaan,

If you are using datamodels they also use $SPLUNK_HOME path as default. Since they use a separate volume name (volume:_splunk_summaries) these can consume your 190GB missing space.

If this reply helps you an upvote is appreciated.

[rlaan]

Thanks for the heads up, i don't believe i have configured anything from that, i have attempted the volume implementation just using the indexers.conf file that is pushed out from deployment server. Can 1 path location exist within 2 volumes?
ex. volume:_splunk_summaries and the one i tried to create ?