I want to achieve 1 day retention for indexed data. How can I achieve this? I have a cluster setup with RF=3 and SF=3. As far as my understanding, I can set frozenTimePeriodInSecs = 86400 , which is equivalent to 1 day? I have the ff configuration on my master indexes.conf .
[testindex] repFactor = auto homePath = $SPLUNK_HOME/var/lib/splunk/testindex/db/ coldPath = $SPLUNK_HOME/var/lib/splunk/testindex/colddb/ thawedPath = $SPLUNK_HOME/var/lib/splunk/stestindex/thaweddb/ coldToFrozenDir = $SPLUNK_HOME/var/lib/splunk/archived/testindex frozenTimePeriodInSecs = 86400
Does it achieved the 1 day retention?
Keep in mind that splunk stores data in buckets, and these contain more than one event. Also, buckets go from hot to warm, then to cold and then frozen - never from hot to frozen. Thus if your buckets are only filled with very few events per day, they might still be written to after several days (i.e., they are still hot), and your maximum age setting doesn't remove the bucket right away. Also, your setting has to apply to all events in a bucket, so your buckets will only get deleted one day after they are no longer being written to.
PS: Alternatively, see the
maxHotSpanSecs setting here as a more precise method to roll your hot buckets.
This line made me jump out of my seat " * CARELESSNESS IN SETTING THIS MAY LEAD TO PERMANENT BRAIN DAMAGE OR LOSS OF JOB." Im a splunk newbie. So it means I cannot set based on number of days?
So what if I set my maxDataSize = 100 ? I have an average of 150MB a day. Probably I can adjust this to 1 week of before it get deleted.
That particular line refers to two other settings,
indexThreads - we're not touching those.
Since buckets rotate based on both size and age, you can use whichever method suits your needs. Since I don't know what your reasons are for deleting data after just one day, you'll have to decide whether to set
maxHotSpanSecs to 86400 so that hot buckets always roll to warm buckets after one day (and, together with your setting of
frozenTimePeriodInSecs = 86400 become deleted a day after that), or whether you can get a desired behavior with
maxDataSize as well - there's no real drawback on either of them.
Just want to make it clear, this config will delete the index data for testindex after 1 day? It will not affect the other index configured right?
repFactor = auto
homePath = $SPLUNKHOME/var/lib/splunk/testindex/db/
coldPath = $SPLUNKHOME/var/lib/splunk/testindex/colddb/
thawedPath = $SPLUNKHOME/var/lib/splunk/stestindex/thaweddb/
coldToFrozenDir = $SPLUNKHOME/var/lib/splunk/archived/testindex
maxHotSpanSecs = 86400
frozenTimePeriodInSecs = 86400
My reason is,I only want to monitor and alert in real time and I don't want to consume more disk resource for this one.
Yes, with those settings you just posted, your buckets will move from hot to warm after one day, and they will get deleted a day after that (i.e., as soon as the most recent event in that bucket is one day old as specified by
These settings apply to your index
testindex as indicated by the [testindex] stanza above the settings. If you wanted them to apply to every index (which you don't!), then you'd have to set them under the [default] stanza.
Now that you said your reason to remove data is because you need the disk space, you might have been better off with the
coldPath.maxDataSizeMB - that would have given you a reliable way to determine how much space your data needs. This method now ensures your data is a maximum of two days old, but depending on how much data you indexed in those two days the size of your index might vary. But for two days, this is probably neglegible.
Don't forget to officially accept @jeffland's answer by clicking "Accept" directly below his answer. This will resolve the post instead of it floating around on Answers as not having an accepted answer. Also, don't forget to upvote users who have helped you find your solution. Thanks!