Understanding Indexes.conf

[alvaroveiga]

Hello guys,
I would like to understand if i have any misconfiguration on my indexes files, and for how long do i keep logs online, archived and when they are deleted (since my HDD is getting full quickly):

[default]
suppressBannerList = 
frozenTimePeriodInSecs = 15778463
throttleCheckPeriod = 15
quarantineFutureSecs = 2592000
partialServiceMetaPeriod = 0
serviceOnlyAsNeeded = true
maxHotBuckets = 3
enableOnlineBucketRepair = true
bucketRebuildMemoryHint = auto
maxRunningProcessGroups = 8
maxDataSize = auto
maxWarmDBCount = 300
assureUTF8 = false
maxHotIdleSecs = 0
enableRealtimeSearch = true
serviceMetaPeriod = 25
repFactor = 0
maxConcurrentOptimizes = 3
maxHotSpanSecs = 7776000
maxTimeUnreplicatedWithAcks = 60
syncMeta = true
coldToFrozenDir = 
maxRunningProcessGroupsLowPriority = 1
serviceSubtaskTimingPeriod = 30
quarantinePastSecs = 77760000
rawChunkSizeBytes = 131072
sync = 0
maxBucketSizeCacheEntries = 1000000
coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/bin/coldToFrozen.py"
rotatePeriodInSecs = 60
memPoolMB = auto
defaultDatabase = main
enableDataIntegrityControl = true
minRawFileSyncSecs = disable
compressRawdata = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
maxTotalDataSizeMB = 500000
maxTimeUnreplicatedNoAcks = 300

[_audit]
coldPath = $SPLUNK_DB/audit/colddb
homePath = $SPLUNK_DB/audit/db
thawedPath = $SPLUNK_DB/audit/thaweddb

[_internal]
frozenTimePeriodInSecs = 2419200
homePath = $SPLUNK_DB/_internaldb/db
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
maxDataSize = 100
coldPath = $SPLUNK_DB/_internaldb/colddb

[_thefishbucket]
frozenTimePeriodInSecs = 2419200
homePath = $SPLUNK_DB/fishbucket/db
thawedPath = $SPLUNK_DB/fishbucket/thaweddb
maxDataSize = 10
coldPath = $SPLUNK_DB/fishbucket/colddb

[history]
frozenTimePeriodInSecs = 604800
homePath = $SPLUNK_DB/historydb/db
thawedPath = $SPLUNK_DB/historydb/thaweddb
maxDataSize = 10
coldPath = $SPLUNK_DB/historydb/colddb

[main]
maxDataSize = auto_high_volume
homePath = $SPLUNK_DB/defaultdb/db
maxHotBuckets = 10
coldPath = $SPLUNK_DB/defaultdb/colddb
maxHotIdleSecs = 86400
maxConcurrentOptimizes = 6
thawedPath = $SPLUNK_DB/defaultdb/thaweddb

[splunklogger]
coldPath = $SPLUNK_DB/splunklogger/colddb
disabled = true
homePath = $SPLUNK_DB/splunklogger/db
thawedPath = $SPLUNK_DB/splunklogger/thaweddb

[summary]
coldPath = $SPLUNK_DB/summarydb/colddb
homePath = $SPLUNK_DB/summarydb/db
thawedPath = $SPLUNK_DB/summarydb/thaweddb

[horsefez]

This looks like an exact copy of the default indexes conf with some added/changed values. And you seem to not know what you are doing.

Anyway. So... I'm assuming you are currently storing all your data in the "main" index.
This means that here the [default] frozenTimePeriodInSecs = 15778463 applies to the retention time. Which is approx. 182 days.

How to fix this:
go to the $SPLUNK_HOME directory (under linux it's /opt/splunk/)
Navigate from there to /opt/splunk/etc/system/local/
Create a file called "indexes.conf"

Write the following:

[main]
frozenTimePeriodInSecs = 604800

Save and restart splunk. Now the data in the main index will be saved for only 7 days instead of 182.

If you wanna know more about what indexes.conf does and what the parameters do, look here:
https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Indexesconf

[ddrillic]

@alvaroveiga, please keep in mind that the twin configuration parameter of frozenTimePeriodInSecs is maxTotalDataSizeMB which as we can see on line #39 has the default of 500000 MBs, around 1/2 TB.

Together they control the size of the index.

[somesoni2]

Read these post about how Splunk's data rentention policy works and what all indexes.conf parameters are used in setting them. Once you know about how it's implemented, you'd be able to read and understand your indexes.conf values.
https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Setaretirementandarchivingpolicy
https://wiki.splunk.com/Deploy:BucketRotationAndRetention