Why is my recursive copy failing on all indexes and preventing all buckets from moving to cold storage and filling up the warm storage faster?

[CooperInvestors]

I recently started receiving the following error on all indexes. In turn, this prevents all buckets from moving to cold storage, Thus filling up the faster warm storage.

ERROR BucketMover - aborting move because recursive copy from src='\_introspection\db\db_1517868285_1517867401_72' to dst='\\\Splunk\System\_introspection\colddb\inflight-db_1517868285_1517867401_72' failed (reason='There are no more files.')

[CooperInvestors · Accepted solution]

After much deliberation, the problem has been resolved.

It turned out to be a permissions issue on the Cold bucket storage.

The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.

To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.