Getting Data In

Is there a way to get the current retention and archiving policies that are defined for backups?

Communicator

Hello Experts,

Is there a way to get the current policies that are defined for backups?
How often/when does the index data move from hot db to warm db
How often/when does the index data move from warm db to cold db
How often/when does the index data move from cold db to frozen and removed all together

We are trying to make sure index data is archived indefinitely and not removed at all due to compliance purposes.

Thanks,

0 Karma
1 Solution

Motivator

Yes you can get the current policies that are defined for backups by reading your indexes.conf configuration file
For more information, Read here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Setaretirementandarchivingpolicy

For other information concerning hot, warm ....buckets, start here http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Backupindexeddata

View solution in original post

Revered Legend

There is great app to monitor Splunk's Health. See this https://splunkbase.splunk.com/app/1919/

You've a dashboard "Available Indexes" in this app which can provide you list of indexes with its current Retention period.

This dashboard is based on the result of Splunk REST Api endpoint for Indexes. Use following query to get more detailed information about the your Splunk indexes.

| rest /services/data/indexes

Communicator

Many thanks to the link to the app and the command.

0 Karma

Communicator

Also executing btool on the indexes configuration with the --debug flag will show which indexes.conf file is used in setting these retention attributes:

./Splunk btool indexes list --debug

This can be redirected to a txt file for additional analysis in a text editor like vim.

Communicator

Thanks. This is interesting data.

0 Karma

Motivator

Yes you can get the current policies that are defined for backups by reading your indexes.conf configuration file
For more information, Read here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Setaretirementandarchivingpolicy

For other information concerning hot, warm ....buckets, start here http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Backupindexeddata

View solution in original post

Communicator

Thanks for these links,
I got the following configured,
My maxDataSize = auto
What is the auto setting?
maxWarmDBCount = 300 ; so that means I can have 300 warm buckets before it is moved to frozen by default

My maxTotalDataSizeMB = 500000 ; I'm assuming this is not equal to maxDataSize

0 Karma

Motivator

1- Here is the syntaxe for maxDataSize :

maxDataSize = |auto|auto_high_volume
  • This is The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
  • Specifying "auto" or "auto_high_volume" will cause Splunk to autotune this parameter (recommended).
  • You should use "auto_high_volume" for high-volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.
  • Defaults to "auto", which sets the size to 750MB.

2- Here is the syntaxe for maxWarmDBCount:

maxWarmDBCount =<nonnegative integer>

This is the maximum number of warm buckets.
- Warm buckets are located in the for the index.
- If set to zero, it will not retain any warm buckets (will roll them to cold as soon as it can)
- Defaults to 300.
- Highest legal value is 4294967295

3-Here is the syntaxe for maxTotalDataSizeMB :

maxTotalDataSizeMB = <nonnegative integer>
  • This is the maximum size of an index (in MB).
  • If an index grows larger than the maximum size, the oldest data is frozen.
  • This parameter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets.
  • Defaults to 500000.
  • Highest legal value is 4294967295

For more informations, start reading here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Indexesconf

0 Karma

Communicator

Thanks for the detailed response. The links are of super help.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!