topic inflight file clean up in Deployment Architecture

inflight file clean up

huynha — Fri, 30 Oct 2020 16:56:23 GMT

In my indexer cluster, one of my indexers has inflight files in the cold and warm storage that range from 1.5-2 months old. There doesn't seem to be any inflight files newer than these and these directories/files are not empty. One index has old inflight files coming out to 1.8TiB of storage.

1.) Why are these files still sticking around for so long? Why isn't Splunk cleaning them up?

2.) How can I get splunk to take care of these files (i.e. delete them)?

Re: inflight file clean up

richgalloway — Fri, 30 Oct 2020 20:13:36 GMT

What do you mean by "inflight files" and where are you finding them?

Re: inflight file clean up

huynha — Sat, 31 Oct 2020 15:07:00 GMT

they are directories that start with "inflight"

e.g.: inflight-db_1598511485_1598506086_416586130_437D3394-A537-4808-90AC-79857895456D

All I know about I learned from this blog post from 2012 https://www.splunk.com/en_us/blog/tips-and-tricks/restoring-an-index.html,

What is an in-flight bucket? When Splunk transitions a bucket from warm to cold, it is considered to be “in-flight”. There is the potential scenario where a bucket has not completely transitioned, specifically if the storage crashed during the move process. You can find these buckets by looking for “in-flight” within the text of the directory name. You should remove the in-flight bucket as well as include this bucket ID as one that must be also copied to the primary storage from the back up location.

NOTE: If you get to this point, it is best to consult with support as you may have edge conditions that warrant modifications to these instructions.

I'm finding them in the cold and warm paths

Re: inflight file clean up

huynha — Tue, 03 Nov 2020 17:50:39 GMT

Got an update from Splunk Support so figured I'd share here:

inflight buckets are actually searchable buckets. There are 2 possible ways to deal with them.

Deleting them is an option because they consume space and won't be rolled out. If you consider they are not needed, go ahead and remove them.

I wouldn't suggest deleting them because they might still hold relevant data. In that case, there can be repaired so they roll out to cold on their own, and eventually age out normally. I'll include instructions on how to fix them:

How to repair buckets:

1- Identify the inflight bucket. e.g inflight-db_1281863647_1281646469_455.

2- If you are in a cluster environment, put the CM in maintenance mode and issue ./splunk offline on the IDX where the inflight bucket is located. You need to find the primary inflight bucket(inflight-db).

3- Rename the inflight bucket to a normal bucket. e.g from inflight-db_1281863647_1281646469_455 to db_1281863647_1281646469_455, then turn back up the IDX and remove the maintenance mode on the CM.

4- After this, Splunk will replicate the bucket and move it to the coldPath.

Re: inflight file clean up

jrobertson — Fri, 05 Sep 2025 14:54:08 GMT

We recently ran into something similar and had to follow this doc on it. https://splunk.my.site.com/customer/s/article/The-disk-is-getting-completely-filled-up-only-on-one-indexer

However, they do not give any suggestions on how to do it in a timely manner when you have many thousands to deal with.

Basically, maintenance mode, splunk offline, cleanup, start splunk and watch it cleanup.

This is the script I used to remediate ours. Update and use at your own risk:

#! /usr/bin/bash

for FN in $(sudo find <cold_path> -maxdepth 3 -type d -name inflight*)
do
    DEST="$(echo ${FN} | sed -e 's/inflight-//g')"
    RF="$(echo ${FN} | sed -e 's/inflight-./r/g')"
    DF="$(echo ${FN} | sed -e 's/inflight-./d/g')"

    if [ -d "${RF}" ] || [ -d "${DF}" ]
    then
        echo "Dup - ${FN}"
    else
        sudo mv -v ${FN} ${DEST}
        #echo "MOVE - ${FN}"
    fi
done

I also ended up changing it so that the duplicates were just removed right then. We needed the space, and why process it twice.