We have an issue where one of four indexers has filled itself with data in the _internal index. The two possible solutions I have found that seems related is either buckets containing new/future data so rolling to frozen is not being triggered or wrong configuration in indexes.conf. This is not relevant in our case as far as I can see.
From: /opt/splunk/etc/system/default/indexes.conf
[_internal]
homePath = $SPLUNK_DB/_internaldb/db
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
tstatsHomePath = volume:_splunk_summaries/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000
frozenTimePeriodInSecs is 30 days on the faulty indexer as it should. The other indexers have the same settings and have 30 days of _internal data as expected.
Here is some of the values on the oldest bucket that still has state cold:
bucketId = _internal~25~26495DAE-9EE6-434E-85ED-89BDA4221021
endEpoch = 1476799418
guId = 26495DAE-9EE6-434E-85ED-89BDA4221021
index = _internal
modTime = 04/24/2017:10:19:10
path = /splunkdata/splunk_default/_internaldb/colddb/rb_1477233888_1476801890_22_BFAA2ABA-6DFC-4AF4-892A-934EC82DF2A2
startEpoch = 1476801890
state = cold
tsidxState = full
What specifically is supposed to happen when buckets are rolled to frozen, is there a script or something that maybe fails to run? Any pointers on how I can troubleshoot this further?
