We recently ran into something similar and had to follow this doc on it. https://splunk.my.site.com/customer/s/article/The-disk-is-getting-completely-filled-up-only-on-one-indexer However, they do not give any suggestions on how to do it in a timely manner when you have many thousands to deal with. Basically, maintenance mode, splunk offline, cleanup, start splunk and watch it cleanup. This is the script I used to remediate ours. Update and use at your own risk: #! /usr/bin/bash
for FN in $(sudo find <cold_path> -maxdepth 3 -type d -name inflight*)
do
DEST="$(echo ${FN} | sed -e 's/inflight-//g')"
RF="$(echo ${FN} | sed -e 's/inflight-./r/g')"
DF="$(echo ${FN} | sed -e 's/inflight-./d/g')"
if [ -d "${RF}" ] || [ -d "${DF}" ]
then
echo "Dup - ${FN}"
else
sudo mv -v ${FN} ${DEST}
#echo "MOVE - ${FN}"
fi
done I also ended up changing it so that the duplicates were just removed right then. We needed the space, and why process it twice.
... View more