Log Retention Period Change

[DawidM]

I want to set up the retention policy for our logs (18 months). I have edited the indexes.conf to specify 

frozenTimePeriodInSecs

however as per splunk documentation this setting, which we have set up as well:

maxTotalDataSizeMB

takes precedence over frozenTimePeriodInSecs . 

Is there any work around to leave the TotalDataSizeMB set up to a specific value and keep the logs only for 18 months regardless the TotalDataSize? 

Thanks for help
Dawid M

[scelikok]

Hi @DawidM,

Since buckets are sitting on Indexers there is nothing to do on Search Heads, there is no need to restart either. Splunk checks buckets for frozen every 60 seconds by default. It depends on your Indexer I/O performance but it shouldn't take too long. 

Some of your old data may still exist if you have date-time parsing issues. If you are still getting some data which has timestamp from past, they will be in the same bucket with the new ones. Splunk will froze that bucket until its newest data is older than 18 months.

If this reply helps you an upvote is appreciated.

[DawidM]

Thank you for your replay.

Just to clarify here is an example from indexes.conf from master server with frozenTimePeriodinSec at the bottom: 

[demisto_poc]
homePath = volume:ssd/demisto_poc/db
coldPath =  volume:sas/demisto_poc/colddb
thawedPath = /splunk_thawed_data/demisto_poc/thaweddb
homePath.maxDataSizeMB = 1000
maxTotalDataSizeMB = 10000
repFactor = auto
frozenTimePeriodInSecs = 48384000

Could you tell me if there is no correlation between those settings that might cause the issue? 

I deployed this change two days ago and till today there is no effect when I check the oldest log available on the search heads.

Thanks 

[scelikok]

Hi @DawidM,

I don't see anything that may cause the issue. Could you please run and share below search outputs?

Below will show buckets time/size/path info for demisto_poc index. Please run the search using  "All time" timerange;

| dbinspect index=demisto_poc 
| convert ctime(*Epoch) 
| eval rawSizeMB=round(rawSize/1024/1024,2) 
| eval sizeOnDiskMB=round(sizeOnDiskMB,2) 
| table index bucketId startEpoch endEpoch rawSizeMB sizeOnDiskMB state path splunk_server
| addcoltotals rawSizeMB sizeOnDiskMB

 Below will show the frozenTimePeriodinSec setting on Indexers;

| rest /services/properties/indexes/_internal/frozenTimePeriodInSecs

If this reply helps you an upvote is appreciated.

[scelikok]

Hi @DawidM,

Although there is CAUTION message in the docs, there is no precedence between frozenTimePeriodInSecs and maxTotalDataSizeMB settings. Splunk will start freezing the buckets which setting hits first. That is why you should set maxTotalDataSizeMB to a value that bigger than 18 months data.

If this reply helps you an upvote is appreciated. 

[DawidM]

Thanks for clarifying this. I have made a change to the file indexes.conf in the directory /opt/splunk/etc/master-apps/sky_cluster_indexer_base/local on the master server and then I have applied the cluster change with a command:

/opt/splunk/bin/splunk apply cluster-bundle

however there is no affect on search head servers. Should I restart them to start deleting the old logs (5 years)?