Index earliest data still moving after increasing index size.

[gduc]

Hello,

A few days ago I had a problem with an index.
The index_size_max was equal to the index_size, with the default setting in the indexes.conf file.

Here is the request I used:

| rest /services/data/indexes | where disabled = 0 | search NOT title = "_*" | eval currentDBSizeGB = round( currentDBSizeMB / 1024) | where currentDBSizeGB > 0 | table splunk_server title summaryHomePath_expanded minTime maxTime currentDBSizeGB totalEventCount frozenTimePeriodInSecs coldToFrozenDir maxTotalDataSizeMB | rename minTime AS earliest maxTime AS latest summaryHomePath_expanded AS index_path currentDBSizeGB AS index_size totalEventCount AS event_cnt frozenTimePeriodInSecs AS index_retention coldToFrozenDir AS index_path_frozen maxTotalDataSizeMB AS index_size_max title AS index 

On May 14th AM =>
-index_max_size set to 512Go
-index_size = 500Go
-latest data age was "uptodate"
-earliest data age was March 19th - 05:11:30

On May 14th PM =>
-index_max_size set to 1536Go (updated)
-index_size = 509Go
-latest data age was "uptodate"
-earliest data age was March 19th - 05:11:30 (still the same date)

On May 18th AM =>
-index_max_size set to 1536Go
-index_size = 524Go
-latest data age was "uptodate"
-earliest data age was March 19th - 05:11:30 (still the same date)

On May 23th AM =>
-index_max_size set to 1536Go
-index_size = 563Go
-latest data age was "uptodate"
-earliest data age was March 23th - 12:22:28 (not anymore the same date)

On May 26th AM => (today)
-index_max_size set to 1536Go
-index_size = 564Go
-latest data age was "uptodate"
-earliest data age was March 28th - 06:46:27 (not anymore the same date)

Since I've increased the maxTotalDataSizeMB in indexes.conf, I'm still losing the oldest data, but the index is bigger days after days.
I also notice that the earliest data ages are not exactly the same between my 2 indexers in my cluster.

By default I must keep 1 year of data, and parameters are set for, aka " frozenTimePeriodInSecs = 31557600 "

Can anyone help me please?

Thanks a lot.

P.S. Can someone explain to me why this request gives me information for 2 of 3 indexes I've got?
index names are csmsi_supervision_ followed by active, passive or servicenow.
"passive" is missing.

Thanks.

[gduc]

here are the stanzas:

file: $SPLUNK_HOME/etc/slave-apps/csmsi_all_indexes/local/indexes.conf
[csmsi_supervision_active]
coldPath = volume:cold/csmsi_supervision_active/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = volume:live/csmsi_supervision_active/db
maxTotalDataSizeMB = 1536000
thawedPath = /r3c/r3cadmaa/thawed/csmsi_supervision_active/thaweddb
repFactor = auto
frozenTimePeriodInSecs = 31557600

[csmsi_supervision_servicenow]
coldPath = volume:cold/csmsi_supervision_servicenow/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = volume:live/csmsi_supervision_servicenow/db
maxTotalDataSizeMB = 512000
thawedPath = /r3c/r3cadmaa/thawed/csmsi_supervision_servicenow/thaweddb
repFactor = auto
frozenTimePeriodInSecs = 31557600

[csmsi_supervision_passive]
coldPath = volume:cold/csmsi_supervision_passive/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = volume:live/csmsi_supervision_passive/db
maxTotalDataSizeMB = 512000
thawedPath = /r3c/r3cadmaa/thawed/csmsi_supervision_passive/thaweddb
frozenTimePeriodInSecs = 31557600
repFactor = auto

file : $SPLUNK_HOME/etc/system/default/indexes.conf

index specific defaults

maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
minHotIdleSecsBeforeForceRoll = auto
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = gzip
enableTsidxReduction = false
suspendHotRollByDeleteQuery = false
tsidxReductionCheckPeriodInSec = 600
timePeriodInSecBeforeTsidxReduction = 604800

Thanks for helping

[richgalloway]

Please share the indexes.conf stanza for the index in question. Please also share the [default] stanza.