How to edit my data retention policy configuration to delete all data older than two or three weeks?

[mtime24]

Hello,

I'm currently running Splunk Enterprise on version 6.3 in a non clustered environment and I'm having some issues getting my data retention policy to work correctly, I'm sure the issue is with my index file and I'm hoping someone in the community to help out here.

Environment:
Splunk Ent 6.3 with Cisco Networks, Cisco ASA, Cisco UCS, EMC VNX, and Windows Infrastructure app installed
single instance deployment

Problem:
Would like to setup a policy that will delete all data that's older than two weeks or three weeks, whatever stops my server from running out of disk space. I've altered the index file located under C:\Program Files\Splunk\etc\system\local several times with no luck, below is the current index file

Copy of my index file

[msad]
homePath = $SPLUNK_DB/msad/db
coldPath = $SPLUNK_DB/msad/colddb
thawedPath = $SPLUNK_DB/msad/thaweddb
maxDataSize = 1024
maxHotBuckets = 2
frozenTimePeriodInSecs = 604800

[perfmon]
homePath = $SPLUNK_DB/perfmon/db
coldPath = $SPLUNK_DB/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb
maxDataSize = 1024
maxHotBuckets = 2
frozenTimePeriodInSecs = 604800

[winevents]
homePath = $SPLUNK_DB/winevents/db
coldPath = $SPLUNK_DB/winevents/colddb
thawedPath = $SPLUNK_DB/winevents/thaweddb
maxDataSize = 1024
maxHotBuckets = 2
frozenTimePeriodInSecs = 604800

[Richfez]

Two options.

You can adjust individual index sizes for each index with the single

maxTotalDataSizeMB = <nonnegative integer>

directive FOR EACH INDEX.

Or, if you would rather adjust on time before deleting, use

frozenTimePeriodInSecs = <nonnegative integer>

Again, for EACH INDEX.

I ask, though, 15 GB/day, you want 100 GB in use - that's less than one week, not two to three weeks.

So you could decide "I want no index to take more than 10 GB" in which case you would have - sample only, you'll need this line in each index -

[winevents]
...other settings here...
maxTotalDataSizeMB = 10000

Or maybe "I want no index to have data older than 3 days" in which case each index would have

[winevents]
...other settings here...
frozenTimePeriodInSecs = 259200

The other settings for maxdatasize and maxhotbuckets don't really matter a whole lot for this (see important note below) - they're for just when Splunk rolls hot data buckets to warm, but warm is still there taking up space so you achieve little. You can read more about each option by searching for them in the indexes.conf documentation.

What I would do is spend a few minutes recording the data size of each index and how far back it has data for. You only have a few indexes I think so this should only take a few minutes. I'd even open a search and search index=whatever for each to make sure the oldest event isn't an oddity that's way back farther than all the other data. Once I have that information, I could tune each index size for the most benefit. Maybe they're "even" and all drop old stuff off at the same point, but maybe you'll find you can keep a month of data X by dropping only one day off data Y.

IMPORTANT NOTE Buckets get deleted by default when they roll to frozen. But, a bucket won't get rolled to frozen until the entire bucket's contents are older than the cutoff. So, in this way keeping buckets smallish will help that (with other trade-offs in performance) by keeping the granularity smaller. Still, you shouldn't have too much of a problem with this with your existing settings.

[Richfez]

You must restart Splunk for the settings to take effect, but the change will be immediate upon restarting.

[somesoni2]

Try something like this. Restart Splunk instance after making this change. The number for maxWarmDBCount might need to be adjusted, so check you server and count the number of buckets/folders in $SPLUNK_DB/msad/db directory.

[msad]
homePath = $SPLUNK_DB/msad/db
coldPath = $SPLUNK_DB/msad/colddb
thawedPath = $SPLUNK_DB/msad/thaweddb
maxDataSize = 1024
maxHotBuckets = 2
frozenTimePeriodInSecs = 604800
maxWarmDBCount =100

[mtime24]

I could really use anyone's help on this. I read the documentation on this and all that document did was to create more questions for me. Basically I have splunk installed in a single instance environment with the following apps: windows infrastructure, cisco networks, cisco asa, cisco ucs, and emc vnx. I have one C drive with 150gb of total disk space and I would like to allocate 50gb for the OS and 100gb for splunk....how do i go about doing that? I'm currently ingesting 15gb of data a day.

here is my indexes.conf file, could someone please tell me what I have to do so splunk doesn't chew up all of my disk space

"global" params (not specific to individual indexes)

sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
enableRealtimeSearch = true
suppressBannerList =
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10

index specific defaults

maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
enableDataIntegrityControl = false
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries\$_index_name\datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000
journalCompression = gzip

By default none of the indexes are replicated.

repFactor = 0

[volume:_splunk_summaries]
path = $SPLUNK_DB

index definitions

[main]
homePath = $SPLUNK_DB\defaultdb\db
coldPath = $SPLUNK_DB\defaultdb\colddb
thawedPath = $SPLUNK_DB\defaultdb\thaweddb
tstatsHomePath = volume:_splunk_summaries\defaultdb\datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[history]
homePath = $SPLUNK_DB\historydb\db
coldPath = $SPLUNK_DB\historydb\colddb
thawedPath = $SPLUNK_DB\historydb\thaweddb
tstatsHomePath = volume:_splunk_summaries\historydb\datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath = $SPLUNK_DB\summarydb\db
coldPath = $SPLUNK_DB\summarydb\colddb
thawedPath = $SPLUNK_DB\summarydb\thaweddb
tstatsHomePath = volume:_splunk_summaries\summarydb\datamodel_summary

[_internal]
homePath = $SPLUNK_DB_internaldb\db
coldPath = $SPLUNK_DB_internaldb\colddb
thawedPath = $SPLUNK_DB_internaldb\thaweddb
tstatsHomePath = volume:_splunk_summaries_internaldb\datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath = $SPLUNK_DB\audit\db
coldPath = $SPLUNK_DB\audit\colddb
thawedPath = $SPLUNK_DB\audit\thaweddb
tstatsHomePath = volume:_splunk_summaries\audit\datamodel_summary

[_thefishbucket]
homePath = $SPLUNK_DB\fishbucket\db
coldPath = $SPLUNK_DB\fishbucket\colddb
thawedPath = $SPLUNK_DB\fishbucket\thaweddb
tstatsHomePath = volume:_splunk_summaries\fishbucket\datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

this index has been removed in the 4.1 series, but this stanza must be

preserved to avoid displaying errors for users that have tweaked the index's

size/etc parameters in local/indexes.conf.

[splunklogger]
homePath = $SPLUNK_DB\splunklogger\db
coldPath = $SPLUNK_DB\splunklogger\colddb
thawedPath = $SPLUNK_DB\splunklogger\thaweddb
disabled = true

[_introspection]
homePath = $SPLUNK_DB_introspection\db
coldPath = $SPLUNK_DB_introspection\colddb
thawedPath = $SPLUNK_DB_introspection\thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600

[mtime24]

dumb question here, where is the Splunk_DB/msad/db directory located?

currently splunk is installed under C:\Program Files\Splunk

[somesoni2]

If you're not changed the default values, $SPLUNK_DB should be C:\Program Files\Splunk\var\lib\splunk

[mtime24]

ok, so if i'm understanding this correctly, inside the C:\Program Files\Splunk\var\lib\splunk\msad I have four folders so my index file should reflect max hotbuckets = 4?

what's interesting is that the splunk folder located inside C:\Program Files\Splunk\var\lib\ is only 6gb's in size, not sure if this folder should be bigger considering my VM has a 180gb C drive with 56gb free at the moment.

BTW: thanks for all your help