on the index server I have a lot of "inflight" directories. Can I delete these files the index is running out of disk space.
inflight-db_1281863647_1281646469_455/
--1281754080-1281646469-317331.tsidx
--1281863647-1281852120-155270.tsidx
--rawdata/
--splunk-autogen-params.dat
--1281863647-1281647883-43400.tsidx
--optimize.result
--Sources.data
--splunk_optimize_logs/
I've had multiple inflight-db's when I accidentally ran out of storage. The raw-data directory did not copy over so I removed the inflight's from the target, cleared up some space, and performed the transition again.
From Splunk's Blog talking about restoring an Index:
Step 3 – What is an in-flight bucket? When Splunk transitions a bucket from warm to cold, it is considered to be “in-flight”. There is the potential scenario where a bucket has not completely transitioned, specifically if the storage crashed during the move process. You can find these buckets by looking for “in-flight” within the text of the directory name. You should remove the in-flight bucket as well as include this bucket ID as one that must be also copied to the primary storage from the back up location.
Blog post: http://blogs.splunk.com/2012/02/21/restoring-an-index/
Those are directories that contain Splunk index data and are typically being actively written to. If Splunk is not doing anything to these directories, then there is probably something wrong. You should not delete them as you will lose the data within those directories.
If these inflight-db directories still exist when Splunkd is not running, these are considered stale directories and can be safely deleted.
In general, files and directories should not be modified/removed while Splunk is still up and running.