Getting Data In
Highlighted

How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Communicator

Hi splunkers,

I want to achieve 1 day retention for indexed data. How can I achieve this? I have a cluster setup with RF=3 and SF=3. As far as my understanding, I can set frozenTimePeriodInSecs = 86400 , which is equivalent to 1 day? I have the ff configuration on my master indexes.conf .

[testindex]
repFactor = auto
homePath   = $SPLUNK_HOME/var/lib/splunk/testindex/db/
coldPath   = $SPLUNK_HOME/var/lib/splunk/testindex/colddb/
thawedPath = $SPLUNK_HOME/var/lib/splunk/stestindex/thaweddb/
coldToFrozenDir =  $SPLUNK_HOME/var/lib/splunk/archived/testindex
frozenTimePeriodInSecs = 86400

Does it achieved the 1 day retention?

Thanks,

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Champion

Keep in mind that splunk stores data in buckets, and these contain more than one event. Also, buckets go from hot to warm, then to cold and then frozen - never from hot to frozen. Thus if your buckets are only filled with very few events per day, they might still be written to after several days (i.e., they are still hot), and your maximum age setting doesn't remove the bucket right away. Also, your setting has to apply to all events in a bucket, so your buckets will only get deleted one day after they are no longer being written to.

In conclusion, have a look here and here, on the second page especially at the setting maxDataSize which governs how quickly your buckets roll from hot to warm.

PS: Alternatively, see the maxHotSpanSecs setting here as a more precise method to roll your hot buckets.

Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Communicator

This line made me jump out of my seat " * CARELESSNESS IN SETTING THIS MAY LEAD TO PERMANENT BRAIN DAMAGE OR LOSS OF JOB." Im a splunk newbie. So it means I cannot set based on number of days?

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Communicator

So what if I set my maxDataSize = 100 ? I have an average of 150MB a day. Probably I can adjust this to 1 week of before it get deleted.

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Champion

That particular line refers to two other settings, memPoolMB and indexThreads - we're not touching those.

Since buckets rotate based on both size and age, you can use whichever method suits your needs. Since I don't know what your reasons are for deleting data after just one day, you'll have to decide whether to set maxHotSpanSecs to 86400 so that hot buckets always roll to warm buckets after one day (and, together with your setting of frozenTimePeriodInSecs = 86400 become deleted a day after that), or whether you can get a desired behavior with maxDataSize as well - there's no real drawback on either of them.

Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Communicator

Just want to make it clear, this config will delete the index data for testindex after 1 day? It will not affect the other index configured right?

[testindex]
repFactor = auto
homePath = $SPLUNKHOME/var/lib/splunk/testindex/db/
coldPath = $SPLUNK
HOME/var/lib/splunk/testindex/colddb/
thawedPath = $SPLUNKHOME/var/lib/splunk/stestindex/thaweddb/
coldToFrozenDir = $SPLUNK
HOME/var/lib/splunk/archived/testindex
maxHotSpanSecs = 86400
frozenTimePeriodInSecs = 86400

My reason is,I only want to monitor and alert in real time and I don't want to consume more disk resource for this one.

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Champion

Yes, with those settings you just posted, your buckets will move from hot to warm after one day, and they will get deleted a day after that (i.e., as soon as the most recent event in that bucket is one day old as specified by frozenTimePeriodInSecs).

These settings apply to your index testindex as indicated by the [testindex] stanza above the settings. If you wanted them to apply to every index (which you don't!), then you'd have to set them under the [default] stanza.

Now that you said your reason to remove data is because you need the disk space, you might have been better off with the homePath.maxDataSizeMB and coldPath.maxDataSizeMB - that would have given you a reliable way to determine how much space your data needs. This method now ensures your data is a maximum of two days old, but depending on how much data you indexed in those two days the size of your index might vary. But for two days, this is probably neglegible.

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Path Finder

No. You can set on number of days by calculating the frozenTimePeriodInSecs.

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Communicator

Hi merp, yes thanks! As mr Jeff perfectly explained it. 😃

0 Karma
Highlighted

Re: How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

Community Manager
Community Manager

Hi @sympatiko

Don't forget to officially accept @jeffland's answer by clicking "Accept" directly below his answer. This will resolve the post instead of it floating around on Answers as not having an accepted answer. Also, don't forget to upvote users who have helped you find your solution. Thanks!

Patrick

0 Karma