Deployment Architecture
Highlighted

Why is my recursive copy failing on all indexes and preventing all buckets from moving to cold storage and filling up the warm storage faster?

I recently started receiving the following error on all indexes. In turn, this prevents all buckets from moving to cold storage, Thus filling up the faster warm storage.

ERROR BucketMover - aborting move because recursive copy from src='\_introspection\db\db_1517868285_1517867401_72' to dst='\\\Splunk\System\_introspection\colddb\inflight-db_1517868285_1517867401_72' failed (reason='There are no more files.')
0 Karma
Highlighted

Re: Why is my recursive copy failing on all indexes and preventing all buckets from moving to cold storage and filling up the warm storage faster?

After much deliberation, the problem has been resolved.

It turned out to be a permissions issue on the Cold bucket storage.

The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.

To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.

View solution in original post

0 Karma