In my indexer cluster, one of my indexers has inflight files in the cold and warm storage that range from 1.5-2 months old. There doesn't seem to be any inflight files newer than these and these directories/files are not empty. One index has old inflight files coming out to 1.8TiB of storage.
1.) Why are these files still sticking around for so long? Why isn't Splunk cleaning them up?
2.) How can I get splunk to take care of these files (i.e. delete them)?
they are directories that start with "inflight"
All I know about I learned from this blog post from 2012 https://www.splunk.com/en_us/blog/tips-and-tricks/restoring-an-index.html,
What is an in-flight bucket? When Splunk transitions a bucket from warm to cold, it is considered to be “in-flight”. There is the potential scenario where a bucket has not completely transitioned, specifically if the storage crashed during the move process. You can find these buckets by looking for “in-flight” within the text of the directory name. You should remove the in-flight bucket as well as include this bucket ID as one that must be also copied to the primary storage from the back up location.
NOTE: If you get to this point, it is best to consult with support as you may have edge conditions that warrant modifications to these instructions.
I'm finding them in the cold and warm paths