We currently have the situation where some buckets are disappearing from Splunk. So my question is, how do I see what/if buckets were deleted by Splunk (e.g., because of insufficient storage) and maybe a reason for the buckets being deleted?
Personally I think that something on the SAN in the background is weird, but I'd like to check Splunk.
The only time Splunk deletes buckets is when cold data is frozen and you don't have an archive script in place. Deleted buckets are logged to SPLUNKHOME/var/log/splunk/splunkdstdout.log.
Hi - so our issue was that we created a volume like this:
[volume:index] path = $SPLUNK_DB maxVolumeDataSizeMB = 15000000
but maxTotalDataSizeInMB remaind 500000.
What we did now is to increase maxTotalDataSizeInMB to 4294967295 for all relevant indexes.
Thank you !