Getting Data In

Index appears to be truncated prior to Max Index Size

BradL
Path Finder

I have an index "eng_1" that has a max size of 500,000 MB. When I look in SplunkOnSplunk it reports this index to be 25% full, however, the oldest data I can query is about 100 days old. Given this index gets about 5GB/day, that amount of history seems right, but the 25% full seems to imply I should be able to go back much further.

| rest /services/data/indexes | search title="eng_1" | table currentDBSizeMB, splunk_server, maxTotalDataSizeMB, maxWarmDBCount, homePath.maxDataSizeMB, coldPath.maxDataSizeMB

yields:

`
currentDBSizeMB splunk_server maxTotalDataSizeMB maxWarmDBCount homePath.maxDataSizeMB coldPath.maxDataSizeMB

1 index1splunk.au1.domain.net 500000 300 300000 200000
1 index1splunk.br1.domain.net 500000 300 300000 200000
7126 index1splunk.eu1.domain.net 500000 300 300000 200000
134915 index1splunk.us1.domain.net 500000 300 300000 200000
`

So on one hand it seems like the amount of history I see is approximately right given the expected max size and the amount of data I ingest every day, but the metadata and SplunkOnSplunk doesn't show that this index is full.

How can I confirm whether the daily incoming data is causing the old data to get evicted?

0 Karma
1 Solution

BradL
Path Finder

Turns out what's happening is the VOLUME as a whole is reaching it's limit. Therefore the indexes residing on the volume get trimmed to keep the volume size in check, while the index itself is only partially full.

View solution in original post

0 Karma

BradL
Path Finder

Turns out what's happening is the VOLUME as a whole is reaching it's limit. Therefore the indexes residing on the volume get trimmed to keep the volume size in check, while the index itself is only partially full.

View solution in original post

0 Karma

somesoni2
Revered Legend

The data retention of an index is influenced by two factors
maxTotalDataSizeMB - If total size reaches beyond this limit, data buckets will be rolled over to frozen
frozenTimePeriodInSecs -= if latest time for a bucket is older than this period, that data bucket will be rolled over to frozen.

Run the following query and check the value for frozenTimePeriodInSecs if it has been reduced (default is 6 years) to lower values like 100 days in your case.

| rest /services/data/indexes | search title="eng_1" | table frozenTimePeriodInSecs 

BradL
Path Finder

Thanks for the reply. I ran the query:

frozenTimePeriodInSecs = 31,536,000 // 1 year = ( 365 x 24 x 3600 )

This is actually what I was expecting.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!