Getting Data In

Logs retention policy

SS1
Path Finder

Hi SPlunkers,

We have multiple sources reporting to same index, what we observe is for few sources we can see the searchable earliest event from the month of September and for few sources the earliest event from August. Does different source have different retention policies? or the data is stored in different buckets which are searchable

 

Retention policy for the index:

Setting Value

maxTotalDataSizeMB307200frozenTimePeriodInSecs188697600homePath.maxDataSizeMB0coldPath.maxDataSizeMB0
Labels (3)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Have a look at very useful Splunk documentation about Splunk retention policy here: https://docs.splunk.com/Documentation/Splunk/8.0.6/Indexer/Setaretirementandarchivingpolicy#Freeze_d...

Total Index size does cause older data to be frozen, so that should be the issue here as retention period is quite large.

0 Karma

SS1
Path Finder

Thanks for the reply.

Could this happen because of the index size which is why the logs older than september are not searchable?  we are able to view fluentd logs in the same index

We currently have a 300GB index, which saves about 3 weeks given current ingestion rates. 

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The retention policy applies at index level. All the sources/sourcetypes within that index will have same retention period.

Having said that, the retention period of N days means that data will be completely available for at least N days. Data older than N days may be seen but they're undergoing frozen process. The reason for partial data past retention period is how Splunk stores and freezes data. Data is stored in buckets and a bucket is only frozen when all events in that bucket are older than retention period. Till that happens data from that bucket will be searchable (hence partial data).

Your retention period is set to 188697600 (2184 days), so you should see all the data for this years Aug/Sep. It could happen that different sources started sending data to Splunk at different time, hence the oldest data you see have different dates for different sources.

0 Karma

SS1
Path Finder

Thanks for the reply.

Could this happen because of the index size which is why the logs older than september are not searchable?  we are able to view fluentd logs in the same index

We currently have a 300GB index, which saves about 3 weeks given current ingestion rates. 

Is there any other setting we need to look into it?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...