Getting Data In

_internal index data not archiving/deleting after 30 days.

Robbie1194
Communicator

Hi guys,

I was wondering if anyone knows why my _internal index information is not archiving/deleting from frozen after 30 days

It wont let me attach a screenshot but in the DMC it shows that the "Data Age vs Frozen Data (days)" is 103/30... Which isn't right!

I can see that the value of frozenTimePeriodInSecs in system/default/indexes.conf is 2592000 (30 days) and using btool shows that the value is being taken but I don't know why it isn't working? Any ideas?

I was thinking of making a new app for config and change it to 31 days to see if it changes anything? Does anyone think this would work? I'm in a clustered environment so I'm a bit worried to make any changes in case it makes it worse!

Any help will be appreciated.

Cheers!

0 Karma

mattymo
Splunk Employee
Splunk Employee

check out the | dbinspect command to examine the buckets in the index. As Teunlaan commented, Splunk will only freeze a bucket once the LATEST event eclipses the frozenTimeInSecs.

You can use dbinspect and a little eval magic to convert the earliest and latest event time to confirm the timespan your buckets cover.

https://answers.splunk.com/answers/112500/dbinspect-fields-names-and-format-changed-in-6.html

If you have low traffic on the box, you will need to tweak indexes.conf for your _internal indexes to set maxHotSpanSecs to something like 86401, to roll it every day, or 604801 to force the bucket to close after 1 week.

Also, if somehow you have future timestamps, this can cause problems with rolling.

- MattyMo

teunlaan
Contributor

What is your bucket size?
It will only delete data if the last message in your bucket is older than 30 day's

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...