Knowledge Management

configuration file for index and summary index

shaganga
New Member

Hi

we need your help in creating the configuration to align the requirements.
we have created index for application logs rpp_pe_idx_dmc and we have created schedule saved search to perform some searches and store the results by enabling summary index at rpp_pe_summary_idx_dmc. Question here is we need to update the indexes.conf to meet below requirements.

  • Hot&Warm buckets will have 90 days of raw data (for index rpp_pe_idx_dmc)
  • Cold buckets will have last 10 months of summary data (for index rpp_pe_summary_idx_dmc)

If we look at my incomplete indexes.conf:

[rpp_pe_idx_dmc]
coldPath = volume:COLD/rpp_pe_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_idx_dmc/thaweddb

[rpp_pe_summary_idx_dmc]
coldPath = volume:COLD/rpp_pe_summary_idx_dmc/colddb
homePath = volume:HOTWARM/rpp_pe_summary_idx_dmc/db
thawedPath = $SPLUNK_DB/rpp_pe_summary_idx_dmc/thaweddb

could you provide us the completed configuration of those two snippets to meet the requirements.

Thanks !!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi shaganga,
let me understand:
you said that retention of rpp_pe_idx_dmc is 90 days in hot/warm data, but what is the retention of cold data?
do you want to use summary only for cold data, not also for hot/warm data? why?
It's not clear your requirement: how long do you want to archive full logs? in other words what is the retention?
How do you want to use summary: for archive summary data or to accelerate searches?

Anyway you can define:

  • the max number oh warm buckets: maxWarmDBCount = ,
  • The maximum size of an index (in MB): maxTotalDataSizeMB = ,
  • Total retention period: frozenTimePeriodInSecs = ,
  • The maximum size in MB for a hot DB to reach before a roll to warm is triggered: maxDataSize = |auto|auto_high_volume,
  • Maximum hot buckets that can exist per index: maxHotBuckets = ,
  • The maximum size of homePath (which contains hot and warm buckets): homePath.maxDataSizeMB = ,
  • The maximum size of coldPath (which contains cold buckets): coldPath.maxDataSizeMB = For full information see http://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Indexesconf.

Bye.
Giuseppe

0 Karma

shaganga
New Member

Hi @Giuseppe

Thanks for quick response. If at all we require to keep raw index for 3 months retention and summary index for 13 months retention. Could you please advise the how configuration looks like?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi shaganga,
to set the retention period of an index you have to put in the related stanzas of your indexes.conf the following row

frozenTimePeriodInSecs = <integer>

So if you have a row index called my_index with a retention of 90 days and a summary index called my_summary with a retention of 13 months (395 days) you have to insert:

[my_index]
frozenTimePeriodInSecs = 7776000

[my_summary]
frozenTimePeriodInSecs = 34128000

obviously remember that a bucket will be deleted when the latest event of the bucket will be out of retention period, so the earliest events of a bucket will remain online more than the retention period.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...