Getting Data In

Is there a way to get the current retention and archiving policies that are defined for backups?

kkossery
Communicator

Hello Experts,

Is there a way to get the current policies that are defined for backups?
How often/when does the index data move from hot db to warm db
How often/when does the index data move from warm db to cold db
How often/when does the index data move from cold db to frozen and removed all together

We are trying to make sure index data is archived indefinitely and not removed at all due to compliance purposes.

Thanks,

0 Karma
1 Solution

stephanefotso
Motivator

Yes you can get the current policies that are defined for backups by reading your indexes.conf configuration file
For more information, Read here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Setaretirementandarchivingpolicy

For other information concerning hot, warm ....buckets, start here http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Backupindexeddata

View solution in original post

somesoni2
Revered Legend

There is great app to monitor Splunk's Health. See this https://splunkbase.splunk.com/app/1919/

You've a dashboard "Available Indexes" in this app which can provide you list of indexes with its current Retention period.

This dashboard is based on the result of Splunk REST Api endpoint for Indexes. Use following query to get more detailed information about the your Splunk indexes.

| rest /services/data/indexes

kkossery
Communicator

Many thanks to the link to the app and the command.

0 Karma

transtrophe
Communicator

Also executing btool on the indexes configuration with the --debug flag will show which indexes.conf file is used in setting these retention attributes:

./Splunk btool indexes list --debug

This can be redirected to a txt file for additional analysis in a text editor like vim.

kkossery
Communicator

Thanks. This is interesting data.

0 Karma

stephanefotso
Motivator

Yes you can get the current policies that are defined for backups by reading your indexes.conf configuration file
For more information, Read here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Setaretirementandarchivingpolicy

For other information concerning hot, warm ....buckets, start here http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Backupindexeddata

kkossery
Communicator

Thanks for these links,
I got the following configured,
My maxDataSize = auto
What is the auto setting?
maxWarmDBCount = 300 ; so that means I can have 300 warm buckets before it is moved to frozen by default

My maxTotalDataSizeMB = 500000 ; I'm assuming this is not equal to maxDataSize

0 Karma

stephanefotso
Motivator

1- Here is the syntaxe for maxDataSize :

maxDataSize = |auto|auto_high_volume
  • This is The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
  • Specifying "auto" or "auto_high_volume" will cause Splunk to autotune this parameter (recommended).
  • You should use "auto_high_volume" for high-volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.
  • Defaults to "auto", which sets the size to 750MB.

2- Here is the syntaxe for maxWarmDBCount:

maxWarmDBCount =<nonnegative integer>

This is the maximum number of warm buckets.
- Warm buckets are located in the for the index.
- If set to zero, it will not retain any warm buckets (will roll them to cold as soon as it can)
- Defaults to 300.
- Highest legal value is 4294967295

3-Here is the syntaxe for maxTotalDataSizeMB :

maxTotalDataSizeMB = <nonnegative integer>
  • This is the maximum size of an index (in MB).
  • If an index grows larger than the maximum size, the oldest data is frozen.
  • This parameter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets.
  • Defaults to 500000.
  • Highest legal value is 4294967295

For more informations, start reading here: http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Indexesconf

0 Karma

kkossery
Communicator

Thanks for the detailed response. The links are of super help.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...