topic Why is the volume not restricting index size? in Getting Data In

Why is the volume not restricting index size?

rlaan — Sat, 23 Jan 2021 05:06:50 GMT

/opt/splunk/etc/deployment-apps/indexer_config/local/indexes.conf [volume:indexer_disk_size] path = $SPLUNK_DB maxVolumeDataSizeMB = 530000 [network] homePath = volume:indexer_disk_size/network/db coldPath = volume:indexer_disk_size/network/colddb thawedPath = $SPLUNK_DB/network/thaweddb maxDataSize = auto_high_volume maxHotIdleSecs = 86400 maxWarmDBCount = 7 frozenTimePeriodInSecs = 7776000 # 1 month = 2592000, 3 month = 7776000

I am experiencing that all of my indexes are able to individually go to a max of 500Gb which i believe is the default value. I am using Splunk 7.3.x. Do i need to change any of my configuration settings, i suspect the volume referencing SPLUNK_DB as the path is causing issues although i have had difficulty.

I have many indexes in this volume that are collectively consuming 720GB of the attempted 530GB configuration (the 530 value was used to differentiate from the baseline 500GB).

Please help me understand what i have done wrong, the configurations are all done on the deployment server in indexes.conf and i am unable to see the configurations or deployment server defined index/volumes on any of my splunk monitoring consoles.

Thanks.

Re: Volume not restricting index size

scelikok — Thu, 21 Jan 2021 10:38:19 GMT

Hi @rlaan,

If you are using datamodels they also use $SPLUNK_HOME path as default. Since they use a separate volume name (volume:_splunk_summaries) these can consume your 190GB missing space.

If this reply helps you an upvote is appreciated.

Re: Volume not restricting index size

rlaan — Thu, 21 Jan 2021 16:37:39 GMT

Thanks for the heads up, i don't believe i have configured anything from that, i have attempted the volume implementation just using the indexers.conf file that is pushed out from deployment server. Can 1 path location exist within 2 volumes?
ex. volume:_splunk_summaries and the one i tried to create ?

Re: Volume not restricting index size

scelikok — Thu, 21 Jan 2021 18:47:22 GMT

Hi @rlaan,

Yes, the same path can be defined on different volumes. Can you please share output of below query?

| rest /services/data/index-volumes | fields title, total_size, max_size, volume_path | eval total_size_gb = if(isnull(total_size), "-", round(total_size / 1024, 2)) | eval max_size_gb = if(isnull(max_size) OR max_size = "infinite", "unlimited", round(max_size / 1024, 2)) | eval disk_usage_gb = total_size_gb." / ".max_size_gb | fields title, disk_usage_gb, max_size_gb, volume_path | rename title as Volume, disk_usage_gb as "Volume Usage (GB)", max_size_gb as "Volume Capacity (GB)", volume_path as "Volume Path"

Re: Volume not restricting index size

rlaan — Fri, 22 Jan 2021 21:06:17 GMT

Here is the output from requested query, i noticed the max size unlimited today in _introspection logs, although am still unsure how to change it, i thought the deployment app pushed to indexers would manage that with indexers.conf

Volume Volume Usage (GB) Volume Capacity (GB) Volume Path

_splunk_summaries

- / unlimited

unlimited

/opt/splunk/var/lib/splunk

My volume wasn't returned in your search, my reference to it being size infinite is from the below _introspection line.

{"datetime":"01-22-2021 13:32:17.668 -0700","log_level":"INFO","component":"Volumes","data":{"name":"indexer_disk_size","max_size":"infinite","volume_path":"/opt/splunk/var/lib/splunk"}}

Re: Volume not restricting index size

scelikok — Sat, 23 Jan 2021 14:29:41 GMT

Hi @rlaan,

If the search does not show your volume, there is something wrong, also the monitoring console is not showing.

Are you using an indexer cluster? If yes these settings should be done on cluster master. I didn't ask at the first time assuming there is no cluster.

Re: Volume not restricting index size

rlaan — Mon, 25 Jan 2021 16:46:05 GMT

We are setup with a following main servers
1- search head
1- deployment/license server
2- heavy forward
2- indexer (no replication)