Getting Data In

Is this 180 day retention policy configuration in indexes.conf appropriate?

sidekix24
Path Finder

Hi,

We are are setting up our indexes to all have a retention policy of 180 total days. 10 days in hot/warm and 170 in cold.

Below is a sample stanza that we plan to setup for each index. Can we get some suggestions if this will fit our needs or if there's a better approach?

[ai_gomez_1]
repFactor=auto
homePath=/opt/splunkdb/ai_gomez_1/db
coldPath=/mnt2/splunkdb-cold/ai_gomez_1/colddb
thawedPath=/mnt2/splunkdb-cold/ai_gomez_1/thaweddb
# Roll hot buckets to warm (after 7-days)
maxHotSpanSecs=604800
# Roll warm buckets to cold when warm bucket count reaches this value (default=300)
maxWarmDBCount=10
# (Delete data) roll warm/cold buckets to frozen bucket (after 6-months)
frozenTimePeriodInSecs=15552000
# Max size of index in MB, max value: 4294967295
maxTotalDataSizeMB=4294967295

Thanks

0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

The transition from hot to warm is simply a transition from read/write buckets to read-only buckets. The transition from WARM to COLD happens based on either reaching the maxWarmDBCount or homePath.maxDataSizeMB. So, you need to use your daily ingest volume to calculate what those settings should be to get about 10 days worth of data kept in HOT/WARM. If you are using auto_high_volume for maxDataSize, your buckets will be ~10GB each. The rest is math. 😉

The transition from COLD to FROZEN happens based on either size constraints or time (frozenTimePeriodInSecs). Note that this is not exact as data ages out by bucket, so the youngest event in any given bucket needs to be older than frozenTimePeriodInSecs to cause the bucket to be rolled to FROZEN.

I would recommend you use the Splunk sizing tool to dial in your settings; it will give you proper indexes.conf settings at the very bottom.

View solution in original post

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

The transition from hot to warm is simply a transition from read/write buckets to read-only buckets. The transition from WARM to COLD happens based on either reaching the maxWarmDBCount or homePath.maxDataSizeMB. So, you need to use your daily ingest volume to calculate what those settings should be to get about 10 days worth of data kept in HOT/WARM. If you are using auto_high_volume for maxDataSize, your buckets will be ~10GB each. The rest is math. 😉

The transition from COLD to FROZEN happens based on either size constraints or time (frozenTimePeriodInSecs). Note that this is not exact as data ages out by bucket, so the youngest event in any given bucket needs to be older than frozenTimePeriodInSecs to cause the bucket to be rolled to FROZEN.

I would recommend you use the Splunk sizing tool to dial in your settings; it will give you proper indexes.conf settings at the very bottom.

0 Karma

sidekix24
Path Finder

Thanks ssievert,

We did use that tool as well and it came up this:

[main]
homePath = volume:hotwarm_cold/defaultdb/db
coldPath = volume:hotwarm_cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
homePath.maxDataSizeMB = 409600
coldPath.maxDataSizeMB = 7372800
maxWarmDBCount = 4294967295
frozenTimePeriodInSecs = 16416000
maxDataSize = auto_high_volume

My question is....it says that this is the setting they came up with assuming all data is going to the main index. We will be using multiple indexes so would we use these same numbers for each index or do we have to break down the numbers using the daily ingestion of each index?

Thanks

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Since indexes.conf settings apply on a per-index basis, you would want to calculate per daily ingest for all data targeted at a given index.

0 Karma

sidekix24
Path Finder

ok...that's what I figured.

Thanks

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

NP. Please accept my answer if it resolves your question. Thanks!

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...