I recently started receiving the following error on all indexes. In turn, this prevents all buckets from moving to cold storage, Thus filling up the faster warm storage.
ERROR BucketMover - aborting move because recursive copy from src='\_introspection\db\db_1517868285_1517867401_72' to dst='\\\Splunk\System\_introspection\colddb\inflight-db_1517868285_1517867401_72' failed (reason='There are no more files.')
After much deliberation, the problem has been resolved.
It turned out to be a permissions issue on the Cold bucket storage.
The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.
To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.
After much deliberation, the problem has been resolved.
It turned out to be a permissions issue on the Cold bucket storage.
The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.
To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.