Summary index issue - Retaining only approx 3 months of data

[manishmittal12]

I am working for a client and last year we created some report for the purpose of audit and scheduled them to send data into default summary index.

since last month we are observing that all the data is gone and summary is only retaining data for approx 3 month and few days.
1. I checked the indexes.conf there is nothing changed w.r.t to default summary index.
2. index size will allow it to grow 500 GB in each indexer.
3. Data age in MC is showing on some indexers its 163 days and while on some its 79 days.
4. on one of the SH event count is very high compared to other SH's.
5. Raw to Index Size Ratio* is 4.39:1
6. current index size is around 130 GB & raw data size is around 600 GB.

The team maintaining platform have already raised the vendor case.

[summary]
homePath = volume:hotwarm/summarydb/db
coldPath = volume:cold/summarydb/colddb
thawedPath = /splunk_data/cold/summarydb/thaweddb
tstatsHomePath = volume:hotwarm/summarydb/datamodel_summary

volume:hotwarm]
path = /splunk_data/hotwarm

4.xTB volume allocated to each index.

4.xTB Splunk volume leaving some headroom

maxVolumeDataSizeMB = 4000000

[volume:cold]
path = /splunk_data/cold

4.xTB volume allocated to each index.

4.xTB Splunk volume leaving some headroom

maxVolumeDataSizeMB = 4000000

However i am trying to understand all possible scenarios because of which these might have happened.
Suggestions and investigation tips are welcome.

i think these might come down to storage of the disk and high number of event. Any suggestion on further investigation?

[woodcock]

The index must be defined inside of indexes.conf on the Search Head or you will not be able to write to it. Yes, even though it actually exists on the Indexers, you still have to define it on the Search Head. I am not kidding.

[manishmittal12]

i dont think these is an issue, as the existing reports are already sending data to summary index. the issue is with only new reports.
also, these is default index which comes when you install Splunk.

[manishmittal12]

i have another problem with summary index now. The new reports which i have scheduled to send data to summary index are not coming through when i search the index.

however the old reports which are previously schedule are appearing properly when i search the index=summary.

i also used the collect command |collect index=summary. still dont see the data coming in summary index.
from the logs i see file is successfully created.
SummaryIndexProcessor - sid:1570199845.200453_579415AE-4C4B-4904-B196-525AE7046218 Successfully wrote file to '/opt/splunk/var/spool/splunk/e023817d6c017a8c_events.stash_new'.

Any pointers on the issue?

[gcusello]

Hi manishmittal12,
did you checked that there isn't any retention rule (frozenTimePeriodInSecs) on your summary index?
You can check this using btool on Indexer

./splunk cmd btool indexes list --debug > my_indexes.txt

and editing the my_indexes.txt file to understand if there are other configurations outside that indexes.conf.

Then (I'm not sure!) check if you have enableTsidxReduction = True and timePeriodInSecBeforeTsidxReduction = xx.

Bye.
Giuseppe

[manishmittal12]

no these is not about frozenTimePeriodInSecs as we have set that in specific index section only. also all the indexes are only managed through single indexes.conf