We have cluster indexer setup with 5 indexers on separate ESX Servers each with 12TB HDD and 128GB RAM.
The cluster replication factor(RF) is 2 and Search factor(SF) is 1. We have one Job scheduler and search head and forwarder nodes.
Splunk version 6.5.3
Our daily log volume is close to 1TB/day. Out of 1TB log volume 85% of data goes to
abc_transaction indexes (RAW data).
indexes.conf file we have set
frozenTimePeriodInSecs = 3888000 (45 days) for
abc_transaction RAW index.
Even though we have set
frozenTimePeriodInSecs as 45 days, it is able to search only last 10 days of
abc_transaction RAW data.
So to keep the
abc_transaction data available for 45 days, we have gone through this Splunk sizing link and added
maxVolumeDataSizeMB = 5976884(5.7TB) attribute for
[abc_transaction] repFactor=auto coldPath = $SPLUNK_DB/abc_transaction/colddb homePath = $SPLUNK_DB/abc_transaction/db thawedPath = $SPLUNK_DB/abc_transaction/thaweddb maxMemMB = 20 maxConcurrentOptimizes = 6 maxHotIdleSecs = 86400 maxHotBuckets = 10 maxDataSize = auto_high_volume frozenTimePeriodInSecs = 3888000 maxVolumeDataSizeMB = 5976884
The following are my questions:
abc_transactiondata, as per the Splunk sizing link which I mentioned above the
maxVolumeDataSizeMB = 11.4TBwhich is almost the disk size of the indexer, is it possible to set the entire disk size as
Any help would be greatly appreciated.
@balachandar, the big twin is -
maxTotalDataSizeMB = 1400000 frozenTimePeriodInSecs= 7776000
Retention, in this case, is 3 months and total size at around 1.4 TBs.
Your estimation of 5.7T is pretty close. I'd include a little extra room just in case and let your frozen time period take care of trimming the index. maxTotalDataSizeMB is the right one to use here as you don't seem to be using volume management on this index.