Getting Data In

How do I set retention for an index to 30 days?

melonman
Motivator

Hi,

I am trying to configure retention period for an index.

I want to archive (compress) the indexed data after 3 months.
I want to delete the indexed data after 1 year.

for test, I set frozenTimePeriodInSecs to 600, but there is noting even in warmDB, coldDB, and frozenDB.

I have been reading documentations, but I still can not get exactly how to set retantion period in indexes.conf.

Could anyone please point me to this information?

Thank you,

Tags (1)
0 Karma
1 Solution

rgonzale6
Path Finder

Does your indexes.conf clause for your index resemble something like this?

[whatever]
coldPath = *path*
homePath = *path*
thawedPath = *path*
coldToFrozenDir = *path*
frozenTimePeriodInSecs = 600

If you do not set a coldToFrozenDir in the indexes.conf clause once the frozenTimePeriodInSecs value is reached splunk simply deletes the data.

View solution in original post

melonman
Motivator

I read this answer, but this does not solve my question...
Is there NO way to specify the exact retantion period by Splunk???

http://splunk-base.splunk.com/answers/2392/how-can-i-rotate-all-hot-and-warm-buckets-older-than-30-d...

0 Karma

rgonzale6
Path Finder

Does your indexes.conf clause for your index resemble something like this?

[whatever]
coldPath = *path*
homePath = *path*
thawedPath = *path*
coldToFrozenDir = *path*
frozenTimePeriodInSecs = 600

If you do not set a coldToFrozenDir in the indexes.conf clause once the frozenTimePeriodInSecs value is reached splunk simply deletes the data.

melonman
Motivator

Now I have this, and see how this configuration works.
Some answers mentioned rebooting splunk will affect the number of warm buckets, so interested in the result.

maxDataSize = 1024 (bucket size is to be 1GB, or gib enough to hold 1 day indexing volume)
maxHotIdleSecs = 86400 (1 day, for hot to warm roll or call the holl-hot-bucket script)
maxWarmDBCount = 30 (30 buckets = 30days, for warm to cold)
frozenTimePeriodInSecs = 7776000 (90 days in sec, cold to frozen)
coldToFrozenDir = /archive/myindex ( after 90 days, index goes here)

Thank you all for helping me out!

adamw
Communicator

You can manually roll hot buckets to warm if you'd like

$SPLUNK_HOME/bin/splunk _internal call /data/indexes/main/roll-hot-buckets -auth admin:password

0 Karma

melonman
Motivator

Yes, I rebooted, but no luck. I think hotDB is not controlled by fronTmePeriodInSecs. Also noticed that Splunk's bucket mover is triggered when:
- backetsize is exceeded (hot to warm)
- backet number is excceeded (warm to cold)
- backet age is exceeded (cold to frozen/delete)

Seems very hard to set exactly 3 month retention.

0 Karma

rgonzale6
Path Finder

Did you restart splunk after making your indexes.conf changes? Do you see anything in your splunkd.log about 'attempting to freeze'?

0 Karma

melonman
Motivator

Yes, I actually set the values same as your example with correct paths. However, there is still nothing in coldToFrozenDir after 600+ sec. Data is continuously generated and indexed. Any idea?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...