Rolling from Warm to Cold, buckets are deleted

[seshi]

We have been using Splunk for about a year or so now and our latest event is always 3 months old. What seems to be happening is that when the warm buckets are being rolled to cold, the buckets are being deleted. For example, the colddb folder for the main index is empty.

Below is the indexes.conf for one of the cluster slaves. None of the defaults have been explicitly modified. Does the default configuration delete the buckets when the roll from warm to cold?

#   Version 6.5.0
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local  
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
#
# This file configures Splunk's indexes and their properties.
#

################################################################################
# "global" params (not specific to individual indexes)
################################################################################
sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
enableRealtimeSearch = true
suppressBannerList = 
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10
rtRouterThreads = 0
rtRouterQueueSize = 10000

################################################################################
# index specific defaults
################################################################################
maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript = 
coldToFrozenDir = 
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
minHotIdleSecsBeforeForceRoll = auto
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript= 
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = gzip
enableTsidxReduction = false
suspendHotRollByDeleteQuery = false
tsidxReductionCheckPeriodInSec = 600
timePeriodInSecBeforeTsidxReduction = 604800

#
# By default none of the indexes are replicated.
#
repFactor = 0

[volume:_splunk_summaries]
path = $SPLUNK_DB

[provider-family:hadoop]
vix.mode          = report
vix.command       = $SPLUNK_HOME/bin/jars/sudobash
vix.command.arg.1 = $HADOOP_HOME/bin/hadoop
vix.command.arg.2 = jar
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar
vix.command.arg.4 = com.splunk.mr.SplunkMR
vix.env.MAPREDUCE_USER             =
vix.env.HADOOP_HEAPSIZE            = 512
vix.env.HADOOP_CLIENT_OPTS         = -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.env.HUNK_THIRDPARTY_JARS       = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.10.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.6.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-exec-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-metastore-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-serde-0.12.0.jar
vix.mapred.job.reuse.jvm.num.tasks = 100
vix.mapred.child.java.opts         = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapred.reduce.tasks            = 0
vix.mapred.job.map.memory.mb       = 2048
vix.mapred.job.reduce.memory.mb    = 512
vix.mapred.job.queue.name          = default
vix.mapreduce.job.jvm.numtasks     = 100
vix.mapreduce.map.java.opts        = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.reduce.java.opts     = -server -Xmx512m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.job.reduces          = 0
vix.mapreduce.map.memory.mb        = 2048
vix.mapreduce.reduce.memory.mb     = 512
vix.mapreduce.job.queuename        = default
vix.splunk.search.column.filter    = 1
vix.splunk.search.mixedmode        = 1
vix.splunk.search.debug            = 0
vix.splunk.search.mr.maxsplits     = 10000
vix.splunk.search.mr.minsplits     = 100
vix.splunk.search.mr.splits.multiplier = 10
vix.splunk.search.mr.poll          = 2000
vix.splunk.search.recordreader     = SplunkJournalRecordReader,ValueAvroRecordReader,SimpleCSVRecordReader,SequenceFileRecordReader
vix.splunk.search.recordreader.avro.regex     = \.avro$
vix.splunk.search.recordreader.csv.regex      = \.([tc]sv)(?:\.(?:gz|bz2|snappy))?$
vix.splunk.search.recordreader.sequence.regex = \.seq$
vix.splunk.home.datanode           = /tmp/splunk/$SPLUNK_SERVER_NAME/
vix.splunk.heartbeat               = 1
vix.splunk.heartbeat.threshold     = 60
vix.splunk.heartbeat.interval      = 1000
vix.splunk.setup.onsearch          = 1
vix.splunk.setup.package           = current

################################################################################
# index definitions
################################################################################

[main]
homePath   = $SPLUNK_DB/defaultdb/db
coldPath   = $SPLUNK_DB/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[rgreer]

I noticed that your coldToFrozenDir is currently blank. According to the indexer documentation (http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/Configureindexstorage)

If you don't set either this attribute or coldToFrozenScript, the indexer will just log the bucket's directory name and then delete it once it rolls to frozen.

While you might not be hitting the frozen time limits, you could be hitting the max index size limits. This would cause the data to roll out faster than your expecting.