maxVolumeDataSizeMB not enforced. Splunk is indexing more than this limit

[GersonGarcia]

All,

I have the following configuration on my indexes.conf

[volume:_splunk_summaries]
path = /usr/ssn/splunkDB/hot
maxVolumeDataSizeMB = 5400000

[volume:splunk_hot]
path = /usr/ssn/splunkDB/hot
maxVolumeDataSizeMB = 5400000

[volume:splunk_cold]
path = /usr/ssn/splunkDB/cold
maxVolumeDataSizeMB = 15000000

But for some reason, my FS is getting full:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 20027260 2932088 16071172 16% /
tmpfs 66019136 0 66019136 0% /dev/shm
/dev/sda1 204580 272 204308 1% /boot/efi
/dev/sda3 267338488 8004488 245747264 4% /usr/ssn
/dev/sdc1 16910151576 12193755728 3857402392 76% /usr/ssn/splunkDB/cold
/dev/sdb1 6150800784 5272422384 565929072 91% /usr/ssn/splunkDB/hot

It should not go over 85%, correct, or do I need any additional configuration per index?

Thank you,

Gerson Garcia

[GersonGarcia]

@acharlieh

1) Yes, I believe they are referencing the volumes. This is our indexes.conf:

################################################################################
# index specific defaults
################################################################################
maxDataSize = auto
maxWarmDBCount = 600
frozenTimePeriodInSecs = 34128000
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
#blockSignSize = 0 ## removed per 6.3.3 upgrade
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000

#
# By default none of the indexes are replicated.
#
repFactor = 0

[volume:_splunk_summaries]
path = /usr/ssn/splunkDB/hot
maxVolumeDataSizeMB = 5400000

[volume:splunk_hot]
path = /usr/ssn/splunkDB/hot
maxVolumeDataSizeMB = 5400000

[volume:splunk_cold]
path = /usr/ssn/splunkDB/cold
maxVolumeDataSizeMB = 15000000

################################################################################
# index definitions
################################################################################

[main]
homePath = volume:splunk_hot/defaultdb/db
coldPath = volume:splunk_cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume
repFactor = auto

[history]
homePath = volume:splunk_hot/historydb/db
coldPath = volume:splunk_cold/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/historydb/datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath = volume:splunk_hot/summarydb/db
coldPath = volume:splunk_cold/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb
tstatsHomePath = volume:_splunk_summaries/summarydb/datamodel_summary

[_internal]
homePath = volume:splunk_hot/_internaldb/db
coldPath = volume:splunk_cold/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath = volume:splunk_hot/audit/db
coldPath = volume:splunk_cold/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary

[_thefishbucket]
homePath = volume:splunk_hot/fishbucket/db
coldPath = volume:splunk_cold/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb
tstatsHomePath = volume:_splunk_summaries/fishbucket/datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

# Removed after 6.3.3 upgrade per error message
#[_blocksignature]
#homePath = volume:splunk_hot/blockSignature/db
#coldPath = volume:splunk_cold/blockSignature/colddb
#thawedPath = $SPLUNK_DB/blockSignature/thaweddb
#tstatsHomePath = volume:_splunk_summaries/blockSignature/datamodel_summary
#maxDataSize = 1000
#frozenTimePeriodInSecs = 0
#maxTotalDataSizeMB = 0

# this index has been removed in the 4.1 series, but this stanza must be
# preserved to avoid displaying errors for users that have tweaked the index's
# size/etc parameters in local/indexes.conf.
#
[splunklogger]
homePath = volume:splunk_hot/splunklogger/db
coldPath = volume:splunk_cold/splunklogger/colddb
thawedPath = $SPLUNK_DB/splunklogger/thaweddb
disabled = true

[_introspection]
homePath = volume:splunk_hot/_introspection/db
coldPath = volume:splunk_cold/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600

# Splunk app configured indexes

[vmware-beta]
homePath = volume:splunk_hot/vmware-beta/db
coldPath = volume:splunk_cold/vmware-beta/colddb
thawedPath = $SPLUNK_DB/vmware-beta/thaweddb

[vmware]
homePath = volume:splunk_hot/vmware/db
coldPath = volume:splunk_cold/vmware/colddb
thawedPath = $SPLUNK_DB/vmware/thaweddb

[vmware-perf]
homePath = volume:splunk_hot/vmware-perf/db
coldPath = volume:splunk_cold/vmware-perf/colddb
thawedPath = $SPLUNK_DB/vmware-perf/thaweddb

[vmware-inv]
homePath = volume:splunk_hot/vmware-inv/db
coldPath = volume:splunk_cold/vmware-inv/colddb
thawedPath = $SPLUNK_DB/vmware-inv/thaweddb

[vmware-taskevent]
homePath = volume:splunk_hot/vmware-taskevent/db
coldPath = volume:splunk_cold/vmware-taskevent/colddb
thawedPath = $SPLUNK_DB/vmware-taskevent/thaweddb

[os]
homePath = volume:splunk_hot/os/db
coldPath = volume:splunk_cold/os/colddb
thawedPath = $SPLUNK_DB/os/thaweddb
disabled = 0
repFactor = auto
maxTotalDataSizeMB = 1048576

[firedalerts]
coldPath = volume:splunk_cold/firedalerts/colddb
homePath = volume:splunk_hot/firedalerts/db
thawedPath = $SPLUNK_DB/firedalerts/thaweddb
repFactor = auto

[unix_summary]
homePath   = volume:splunk_hot/unix_summary/db
coldPath   = volume:splunk_cold/unix_summary/colddb
thawedPath = $SPLUNK_DB/unix_summary/thaweddb
disabled = 0
repFactor = auto
maxTotalDataSizeMB = 10000

[sos]
homePath = volume:splunk_hot/sos/db
coldPath = volume:splunk_cold/sos/colddb
thawedPath = $SPLUNK_DB/sos/thaweddb
frozenTimePeriodInSecs = 2419200
disabled = 0
repFactor = auto

[sos_summary_daily]
homePath = volume:splunk_hot/sos_summary_daily/db
coldPath = volume:splunk_cold/sos_summary_daily/colddb
thawedPath = $SPLUNK_DB/sos_summary_daily/thaweddb
disabled = 0
repFactor = auto


# Customer Generated Custom Indexes
[ssn_prfm]
coldPath = volume:splunk_cold/ssn_prfm/colddb
homePath = volume:splunk_hot/ssn_prfm/db
thawedPath = $SPLUNK_DB/ssn_prfm/thaweddb
repFactor = auto

[summary_ssn_prfm]
coldPath = volume:splunk_cold/summary_ssn_prfm/colddb
homePath = volume:splunk_hot/summary_ssn_prfm/db
thawedPath = $SPLUNK_DB/summary_ssn_prfm/thaweddb
repFactor = auto

[summary_temp]
coldPath = volume:splunk_cold/summary_temp/colddb
homePath = volume:splunk_hot/summary_temp/db
maxTotalDataSizeMB = 1000
thawedPath = $SPLUNK_DB/summary_temp/thaweddb
repFactor = auto

[sma]
coldPath = volume:splunk_cold/sma/colddb
homePath = volume:splunk_hot/sma/db
maxTotalDataSizeMB = 500000
thawedPath = $SPLUNK_DB/sma/thaweddb
repFactor = auto

[summary_messaging_bus]
coldPath = volume:splunk_cold/summary_messaging_bus/colddb
homePath = volume:splunk_hot/summary_messaging_bus/db
thawedPath = $SPLUNK_DB/summary_messaging_bus/thawddb
repFactor = auto

[ssn_test]
homePath = volume:splunk_hot/ssn_test
coldPath = volume:splunk_cold/ssn_test/colddb
thawedPath = $SPLUNK_DB/ssn_test/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 1000
repFactor = auto

[summary_sourcetypes_ssn]
homePath = volume:splunk_hot/summary_sourcetypes_ssn/db
coldPath = volume:splunk_cold/summary_sourcetypes_ssn/colddb
thawedPath = $SPLUNK_DB/summary_sourcetypes_ssn/thaweddb
repFactor = auto

[summary_loglevel_ssn]
homePath = volume:splunk_hot/summary_loglevel_ssn/db
coldPath = volume:splunk_cold/summary_loglevel_ssn/colddb
thawedPath = $SPLUNK_DB/summary_loglevel_ssn/thaweddb
repFactor = auto

[summary_sources_ssn]
homePath = volume:splunk_hot/summary_sources_ssn/db
coldPath = volume:splunk_cold/summary_sources_ssn/colddb
thawedPath = $SPLUNK_DB/summary_sources_ssn/thaweddb
repFactor = auto

[summary_sourcetype_host_source_ssn]
homePath = volume:splunk_hot/summary_sourcetype_host_source_ssn/db
coldPath = volume:splunk_cold/summary_sourcetype_host_source_ssn/colddb
thawedPath = $SPLUNK_DB/summary_sourcetype_host_source_ssn/thaweddb
repFactor = auto

[uf-fschange]
homePath = volume:splunk_hot/uf-fschange/db
coldPath = volume:splunk_cold/uf-fschange/colddb
thawedPath = $SPLUNK_DB/uf-fschange/thaweddb
repFactor = auto

[ssn]
coldPath = volume:splunk_cold/ssn/colddb
homePath = volume:splunk_hot/ssn/db
thawedPath = $SPLUNK_DB/ssn/thaweddb
maxDataSize = auto_high_volume
homePath.maxDataSizeMB= 3145728
maxTotalDataSizeMB = 14680064
repFactor = auto

[nagios]
homePath = volume:splunk_hot/nagios/db
coldPath = volume:splunk_cold/nagios/colddb
thawedPath = $SPLUNK_DB/nagios/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 500000
repFactor = auto

[oracle]
homePath = volume:splunk_hot/oracle/db
coldPath = volume:splunk_cold/oracle/colddb
thawedPath = $SPLUNK_DB/oracle/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 500000
repFactor = auto

[remedy]
homePath = volume:splunk_hot/remedy/db
coldPath = volume:splunk_cold/remedy/colddb
thawedPath = $SPLUNK_DB/remedy/thaweddb
maxDataSize = auto_high_volume
repFactor = auto

[network]
homePath = volume:splunk_hot/network/db
coldPath = volume:splunk_cold/network/colddb
thawedPath = $SPLUNK_DB/network/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 500000
repFactor = auto

[scratch]
homePath = volume:splunk_hot/scratch/db
coldPath = volume:splunk_cold/scratch/colddb
thawedPath = $SPLUNK_DB/scratch/thaweddb
maxDataSize = auto_high_volume
repFactor = auto

[fschange]
homePath = volume:splunk_hot/fschange/db
coldPath = volume:splunk_cold/fschange/colddb
thawedPath = $SPLUNK_DB/fschange/thaweddb
maxDataSize = auto_high_volume
repFactor = auto

[summary_trap_counts_by_trap_type_sourcetype]
coldPath = volume:splunk_cold/summary_trap_counts_by_trap_type_sourcetype/colddb
homePath = volume:splunk_hot/summary_trap_counts_by_trap_type_sourcetype/db
thawedPath = $SPLUNK_DB/summary_trap_counts_by_trap_type_sourcetype/thaweddb
repFactor = auto

[summary_index_sourcetype_host_source]
coldPath = volume:splunk_cold/summary_index_sourcetype_host_source/colddb
homePath = volume:splunk_hot/summary_index_sourcetype_host_source/db
thawedPath = $SPLUNK_DB/summary_index_sourcetype_host_source/thaweddb
repFactor = auto

[summary_threads_count_by_state]
coldPath = volume:splunk_cold/summary_threads_count_by_state/colddb
homePath = volume:splunk_hot/summary_threads_count_by_state/db
thawedPath = $SPLUNK_DB/summary_threads_count_by_state/thaweddb
disabled = 1
repFactor = auto

[summary_web_services]
coldPath = volume:splunk_cold/summary_web_services/colddb
homePath = volume:splunk_hot/summary_web_services/db
thawedPath = $SPLUNK_DB/summary_web_services/thaweddb
repFactor = auto

[summary_os_timebased_stats]
coldPath = volume:splunk_cold/summary_os_timebased_stats/colddb
homePath = volume:splunk_hot/summary_os_timebased_stats/db
thawedPath = $SPLUNK_DB/summary_os_timebased_stats/thaweddb
repFactor = auto

[summary_dlca]
coldPath = volume:splunk_cold/summary_dlca/colddb
homePath = volume:splunk_hot/summary_dlca/db
thawedPath = $SPLUNK_DB/summary_dlca/thaweddb
repFactor = auto

[ucs]
coldPath = volume:splunk_cold/ucs/colddb
homePath = volume:splunk_hot/ucs/db
maxTotalDataSizeMB = 200000
thawedPath = $SPLUNK_DB/ucs/thaweddb
repFactor = auto

[security]
coldPath = volume:splunk_cold/security/colddb
homePath = volume:splunk_hot/security/db
maxTotalDataSizeMB = 500000
thawedPath = $SPLUNK_DB/security/thaweddb
repFactor = auto

[ssn-ext]
coldPath = volume:splunk_cold/ssn-ext/colddb
homePath = volume:splunk_hot/ssn-ext/db
maxTotalDataSizeMB = 500000
thawedPath = $SPLUNK_DB/ssn-ext/thaweddb
repFactor = auto

[itil]
coldPath = volume:splunk_cold/itil/colddb
homePath = volume:splunk_hot/itil/db
thawedPath = $SPLUNK_DB/itil/thaweddb
maxDataSize = auto_high_volume
homePath.maxDataSizeMB = 5000
maxTotalDataSizeMB = 25600
repFactor = auto

[windows]
homePath = volume:splunk_hot/windows/db
coldPath = volume:splunk_cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 75000
maxTotalDataSizeMB = 150000
repFactor = auto

[wineventlog]
homePath = volume:splunk_hot/wineventlog/db
coldPath = volume:splunk_cold/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 25000
maxTotalDataSizeMB = 75000
repFactor = auto

[perfmon]
homePath = volume:splunk_hot/perfmon/db
coldPath = volume:splunk_cold/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 50000
maxTotalDataSizeMB = 100000
repFactor = auto

[sro]
coldPath = volume:splunk_cold/sro/colddb
homePath = volume:splunk_hot/sro/db
thawedPath = $SPLUNK_DB/sro/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 5000
maxTotalDataSizeMB = 25600
repFactor = auto

[mysql]
homePath = volume:splunk_hot/mysql/db
coldPath = volume:splunk_cold/mysql/colddb
thawedPath = $SPLUNK_DB/mysql/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 75000
maxTotalDataSizeMB = 150000
repFactor = auto

[ssn-infra]
coldPath = volume:splunk_cold/ssn-infra/colddb
homePath = volume:splunk_hot/ssn-infra/db
thawedPath = $SPLUNK_DB/ssn-infra/thaweddb
maxDataSize = auto
homePath.maxDataSizeMB = 5000
maxTotalDataSizeMB = 25600
repFactor = auto

[qualys]
coldPath = volume:splunk_cold/qualys/colddb
homePath = volume:splunk_hot/qualys/db
thawedPath = $SPLUNK_DB/qualys/thaweddb
maxDataSize = auto_high_volume
homePath.maxDataSizeMB = 2000
maxTotalDataSizeMB = 10000
repFactor = auto

2) But if it would sum the quota (5400000+5400000=10800000) that is less than 85% of the total, hot=6T and cold=16T (6291456+16777216=23068672), so if Splunk would take both, it should stop at 46%, correct?

3) What will happen if I lower the maxVolumeDataSizeMB to 5100000? The oldest data will be moved to cold and then the cold volume will increase? Should I lower both?

Thank you,
Gerson

[sloshburch]

I wonder if the issue here is actually with the nature of how data rolls. I don't know confidently, but is it possible that you have other indexes using the same path, but not through the volume definition, which is filling this up? In other words, I think the maxVolumeDataSizeMB may not kick in unless data comes in for an index that uses the respective volume. It's kinda like how data doesn't roll out until something needs to take its spot.

Just tossing that out there but obviously, I've not reviewed the conf file with a careful eye. This could be worth opening a support ticket since it may get subtle and since data is so important, we wouldn't want you to make a change, risk your data, all on our simple brains. Fair?

[acharlieh]

From the indexes.conf spec file:

Note that this it will act only on those indexes which reference this volume, not on the total size of the path set in the path attribute of this volume.

Therefore you need to configure your indexes to reference the volume and not just put the indexes to happen to reside in the same path. (You're not showing this configuration, so maybe you are already, but you ask if you need per index configuration so I figured better to not assume)

Secondly, you have two volumes referencing the same path, assuming your indexes are utilizing volume definitions, that means that each part is using its own quota. So 5400000 MB for summaries and 5400000 MB for hot, which would be twice the expected quota.

Third 5400000 MB would 5,529,600,000 1K blocks... That's more than the 5,272,422,384 used that your df output states... If your target is 85% of your disk that would be 5105645 MB... (5100000 for a nice round number). Splunk tends to operate in scales of 1024 not 1000.