topic Re: Need to set an upper size limit across multiple volumes in Getting Data In

Need to set an upper size limit across multiple volumes

jducote — Wed, 13 Feb 2013 17:03:19 GMT

I need to set an upper size limit across multiple volumes, for indexing, regardless of hot/warm/cold status. It is my understanding that I may do this by setting the homePath.maxDataSizeMB switch in indexes.conf. Do I need to establish the three volumes individually, or will setting (in example) homePath.maxDataSizeMB=850000[MB] limit each registered volume as having an upper limit of 850GB? Or do I need to specify the limit for each volume, in the same file?

Some background:

3 Volumes of 1TB (986GB usable):
/dev/md127 985G 897G 38G 96% /opt/splunk
/dev/md125 985G 394G 541G 43% /opt/splunk-index
/dev/md126 985G 240G 695G 26% /opt/splunk-index2

As you can see, we are nearly full on the first volume. Pruning index size was successful, but mostly affected /opt/splunk-index. My desired state would be to have data more evenly distributed amongst the three volumes.

Re: Need to set an upper size limit across multiple volumes

yannK — Wed, 13 Feb 2013 17:12:11 GMT

If you want to control the size per volume instead of per index, check this guide
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configureindexstoragesize

The other way is more delicate, you can be very specific for each index using the number of hot and warm buckets to spread the buckets between homePath and coldPath.
http://wiki.splunk.com/Deploy:BucketRotationAndRetention

Re: Need to set an upper size limit across multiple volumes

jducote — Wed, 13 Feb 2013 18:55:48 GMT

Thanks, it looks like using the homePath.maxDataSizeMB=X will be the most elegant option, per:
"Space taken by the whole index
The maximum size of all buckets for an index is maxTotalDataSizeMB. This is the size of hot+warm+cold. If you have hot+warm and cold on different volumes, try to maximize the use of the space." (http://wiki.splunk.com/Deploy:BucketRotationAndRetention)