inflight file clean up

[huynha]

In my indexer cluster, one of my indexers has inflight files in the cold and warm storage that range from 1.5-2 months old.  There doesn't seem to be any inflight files newer than these and these directories/files are not empty.  One index has old inflight files coming out to 1.8TiB of storage.

1.) Why are these files still sticking around for so long? Why isn't Splunk cleaning them up?

2.) How can I get splunk to take care of these files (i.e. delete them)?

[huynha · Accepted solution]

Got an update from Splunk Support so figured I'd share here:

inflight buckets are actually searchable buckets. There are 2 possible ways to deal with them.

Deleting them is an option because they consume space and won't be rolled out. If you consider they are not needed, go ahead and remove them.

I wouldn't suggest deleting them because they might still hold relevant data. In that case, there can be repaired so they roll out to cold on their own, and eventually age out normally.  I'll include instructions on how to fix them:

How to repair buckets:

1- Identify the inflight bucket. e.g inflight-db_1281863647_1281646469_455.

2- If you are in a cluster environment, put the CM in maintenance mode and issue ./splunk offline on the IDX where the inflight bucket is located. You need to find the primary inflight bucket(inflight-db).

3- Rename the inflight bucket to a normal bucket. e.g from inflight-db_1281863647_1281646469_455 to db_1281863647_1281646469_455, then turn back up the IDX and remove the maintenance mode on the CM.

4- After this, Splunk will replicate the bucket and move it to the coldPath.

[richgalloway]

What do you mean by "inflight files" and where are you finding them?

[huynha]

they are directories that start with "inflight"

e.g.: inflight-db_1598511485_1598506086_416586130_437D3394-A537-4808-90AC-79857895456D

All I know about I learned from this blog post from 2012 https://www.splunk.com/en_us/blog/tips-and-tricks/restoring-an-index.html,

What is an in-flight bucket?  When Splunk transitions a bucket from warm to cold, it is considered to be “in-flight”.  There is the potential scenario where a bucket has not completely transitioned, specifically if the storage crashed during the move process.  You can find these buckets by looking for “in-flight” within the text of the directory name.  You should remove the in-flight bucket as well as include this bucket ID as one that must be also copied to the primary storage from the back up location.

NOTE:  If you get to this point, it is best to consult with support as you may have edge conditions that warrant modifications to these instructions.

I'm finding them in the cold and warm paths