Knowledge Management
Highlighted

configuration file for index and summary index

New Member

Hi

we need your help in creating the configuration to align the requirements.
we have created index for application logs rpppeidxdmc and we have created schedule saved search to perform some searches and store the results by enabling summary index at rpppesummaryidx_dmc. Question here is we need to update the indexes.conf to meet below requirements.

  • Hot&Warm buckets will have 90 days of raw data (for index rpppeidx_dmc)
  • Cold buckets will have last 10 months of summary data (for index rpppesummaryidxdmc)

If we look at my incomplete indexes.conf:

[rpppeidxdmc]
coldPath = volume:COLD/rpp
peidxdmc/colddb
homePath = volume:HOTWARM/rpppeidxdmc/db
thawedPath = $SPLUNK
DB/rpppeidx_dmc/thaweddb

[rpppesummaryidxdmc]
coldPath = volume:COLD/rpppesummaryidxdmc/colddb
homePath = volume:HOTWARM/rpppesummaryidxdmc/db
thawedPath = $SPLUNKDB/rpppesummaryidx_dmc/thaweddb

could you provide us the completed configuration of those two snippets to meet the requirements.

Thanks !!

0 Karma
Highlighted

Re: configuration file for index and summary index

Legend

Hi shaganga,
let me understand:
you said that retention of rpppeidx_dmc is 90 days in hot/warm data, but what is the retention of cold data?
do you want to use summary only for cold data, not also for hot/warm data? why?
It's not clear your requirement: how long do you want to archive full logs? in other words what is the retention?
How do you want to use summary: for archive summary data or to accelerate searches?

Anyway you can define:

  • the max number oh warm buckets: maxWarmDBCount = ,
  • The maximum size of an index (in MB): maxTotalDataSizeMB = ,
  • Total retention period: frozenTimePeriodInSecs = ,
  • The maximum size in MB for a hot DB to reach before a roll to warm is triggered: maxDataSize = |auto|autohighvolume,
  • Maximum hot buckets that can exist per index: maxHotBuckets = ,
  • The maximum size of homePath (which contains hot and warm buckets): homePath.maxDataSizeMB = ,
  • The maximum size of coldPath (which contains cold buckets): coldPath.maxDataSizeMB = For full information see http://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Indexesconf.

Bye.
Giuseppe

0 Karma
Highlighted

Re: configuration file for index and summary index

New Member

Hi @Giuseppe

Thanks for quick response. If at all we require to keep raw index for 3 months retention and summary index for 13 months retention. Could you please advise the how configuration looks like?

0 Karma
Highlighted

Re: configuration file for index and summary index

Legend

Hi shaganga,
to set the retention period of an index you have to put in the related stanzas of your indexes.conf the following row

frozenTimePeriodInSecs = <integer>

So if you have a row index called my_index with a retention of 90 days and a summary index called my_summary with a retention of 13 months (395 days) you have to insert:

[my_index]
frozenTimePeriodInSecs = 7776000

[my_summary]
frozenTimePeriodInSecs = 34128000

obviously remember that a bucket will be deleted when the latest event of the bucket will be out of retention period, so the earliest events of a bucket will remain online more than the retention period.

Bye.
Giuseppe

0 Karma