How to configure indexes.conf to have indexed data deleted after 1 day or 24 hours?

[sympatiko]

Hi splunkers,

I want to achieve 1 day retention for indexed data. How can I achieve this? I have a cluster setup with RF=3 and SF=3. As far as my understanding, I can set frozenTimePeriodInSecs = 86400 , which is equivalent to 1 day? I have the ff configuration on my master indexes.conf .

[testindex]
repFactor = auto
homePath   = $SPLUNK_HOME/var/lib/splunk/testindex/db/
coldPath   = $SPLUNK_HOME/var/lib/splunk/testindex/colddb/
thawedPath = $SPLUNK_HOME/var/lib/splunk/stestindex/thaweddb/
coldToFrozenDir =  $SPLUNK_HOME/var/lib/splunk/archived/testindex
frozenTimePeriodInSecs = 86400

Does it achieved the 1 day retention?

Thanks,

[merp96]

You can set on number of days by calculating the frozenTimePeriodInSecs. However this also depends on the bucket size "maxDataSize" set to 750 MB by default in indexes.conf.
If you receive 1 GB per day and your bucket size if set to default 750 MB.
Ist bucket will have 750 MB and second bucket will have the rest 1GB - 750 MB.
next day splunk will only delete bucket one and will not delete bucket two as it has the next days data.
Bottomline :- You must manage maxDataSize and frozenTimePeriodInSecs based on your per day data volume to achieve your retention goal.

[sympatiko]

Hi Im not concern with the data size. Im more concern on the time it keeps on the buckets.

[merp96]

Hi
Then I guess as Jeff as suggested you need to go with maxHotSpanSecs

[jeffland]

Keep in mind that splunk stores data in buckets, and these contain more than one event. Also, buckets go from hot to warm, then to cold and then frozen - never from hot to frozen. Thus if your buckets are only filled with very few events per day, they might still be written to after several days (i.e., they are still hot), and your maximum age setting doesn't remove the bucket right away. Also, your setting has to apply to all events in a bucket, so your buckets will only get deleted one day after they are no longer being written to.

In conclusion, have a look here and here, on the second page especially at the setting maxDataSize which governs how quickly your buckets roll from hot to warm.

PS: Alternatively, see the maxHotSpanSecs setting here as a more precise method to roll your hot buckets.

[merp96]

No. You can set on number of days by calculating the frozenTimePeriodInSecs.

[sympatiko]

Hi merp, yes thanks! As mr Jeff perfectly explained it. 😃

[ppablo]

Hi @sympatiko

Don't forget to officially accept @jeffland's answer by clicking "Accept" directly below his answer. This will resolve the post instead of it floating around on Answers as not having an accepted answer. Also, don't forget to upvote users who have helped you find your solution. Thanks!

Patrick

[sympatiko]

This line made me jump out of my seat " * CARELESSNESS IN SETTING THIS MAY LEAD TO PERMANENT BRAIN DAMAGE OR LOSS OF JOB." Im a splunk newbie. So it means I cannot set based on number of days?

[sympatiko]

So what if I set my maxDataSize = 100 ? I have an average of 150MB a day. Probably I can adjust this to 1 week of before it get deleted.

[jeffland]

That particular line refers to two other settings, memPoolMB and indexThreads - we're not touching those.

Since buckets rotate based on both size and age, you can use whichever method suits your needs. Since I don't know what your reasons are for deleting data after just one day, you'll have to decide whether to set maxHotSpanSecs to 86400 so that hot buckets always roll to warm buckets after one day (and, together with your setting of frozenTimePeriodInSecs = 86400 become deleted a day after that), or whether you can get a desired behavior with maxDataSize as well - there's no real drawback on either of them.

[sympatiko]

Just want to make it clear, this config will delete the index data for testindex after 1 day? It will not affect the other index configured right?

[testindex]
repFactor = auto
homePath = $SPLUNK_HOME/var/lib/splunk/testindex/db/
coldPath = $SPLUNK_HOME/var/lib/splunk/testindex/colddb/
thawedPath = $SPLUNK_HOME/var/lib/splunk/stestindex/thaweddb/
coldToFrozenDir = $SPLUNK_HOME/var/lib/splunk/archived/testindex
maxHotSpanSecs = 86400
frozenTimePeriodInSecs = 86400

My reason is,I only want to monitor and alert in real time and I don't want to consume more disk resource for this one.

[jeffland]

Yes, with those settings you just posted, your buckets will move from hot to warm after one day, and they will get deleted a day after that (i.e., as soon as the most recent event in that bucket is one day old as specified by frozenTimePeriodInSecs).

These settings apply to your index testindex as indicated by the [testindex] stanza above the settings. If you wanted them to apply to every index (which you don't!), then you'd have to set them under the [default] stanza.

Now that you said your reason to remove data is because you need the disk space, you might have been better off with the homePath.maxDataSizeMB and coldPath.maxDataSizeMB - that would have given you a reliable way to determine how much space your data needs. This method now ensures your data is a maximum of two days old, but depending on how much data you indexed in those two days the size of your index might vary. But for two days, this is probably neglegible.