configuration file for index and summary index

[shaganga]

Hi

we need your help in creating the configuration to align the requirements.
we have created index for application logs rpp_pe_idx_dmc and we have created schedule saved search to perform some searches and store the results by enabling summary index at rpp_pe_summary_idx_dmc. Question here is we need to update the indexes.conf to meet below requirements.

• Hot&Warm buckets will have 90 days of raw data (for index rpp_pe_idx_dmc)
• Cold buckets will have last 10 months of summary data (for index rpp_pe_summary_idx_dmc)

If we look at my incomplete indexes.conf:

[rpp_pe_idx_dmc]
coldPath = volume:COLD/rpp_pe_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_idx_dmc/thaweddb

[rpp_pe_summary_idx_dmc]
coldPath = volume:COLD/rpp_pe_summary_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_summary_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_summary_idx_dmc/thaweddb

could you provide us the completed configuration of those two snippets to meet the requirements.

Thanks !!

[gcusello]

Hi shaganga,
let me understand:
you said that retention of rpp_pe_idx_dmc is 90 days in hot/warm data, but what is the retention of cold data?
do you want to use summary only for cold data, not also for hot/warm data? why?
It's not clear your requirement: how long do you want to archive full logs? in other words what is the retention?
How do you want to use summary: for archive summary data or to accelerate searches?

Anyway you can define:

• the max number oh warm buckets: maxWarmDBCount = ,
• The maximum size of an index (in MB): maxTotalDataSizeMB = ,
• Total retention period: frozenTimePeriodInSecs = ,
• The maximum size in MB for a hot DB to reach before a roll to warm is triggered: maxDataSize = |auto|auto_high_volume,
• Maximum hot buckets that can exist per index: maxHotBuckets = ,
• The maximum size of homePath (which contains hot and warm buckets): homePath.maxDataSizeMB = ,
• The maximum size of coldPath (which contains cold buckets): coldPath.maxDataSizeMB = For full information see http://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Indexesconf.

Bye.
Giuseppe

[shaganga]

Hi @Giuseppe

Thanks for quick response. If at all we require to keep raw index for 3 months retention and summary index for 13 months retention. Could you please advise the how configuration looks like?

[gcusello]

Hi shaganga,
to set the retention period of an index you have to put in the related stanzas of your indexes.conf the following row

frozenTimePeriodInSecs = <integer>

So if you have a row index called my_index with a retention of 90 days and a summary index called my_summary with a retention of 13 months (395 days) you have to insert:

[my_index]
frozenTimePeriodInSecs = 7776000

[my_summary]
frozenTimePeriodInSecs = 34128000

obviously remember that a bucket will be deleted when the latest event of the bucket will be out of retention period, so the earliest events of a bucket will remain online more than the retention period.

Bye.
Giuseppe