Deployment Architecture

Why is my recursive copy failing on all indexes and preventing all buckets from moving to cold storage and filling up the warm storage faster?

CooperInvestors
Engager

I recently started receiving the following error on all indexes. In turn, this prevents all buckets from moving to cold storage, Thus filling up the faster warm storage.

ERROR BucketMover - aborting move because recursive copy from src='\_introspection\db\db_1517868285_1517867401_72' to dst='\\\Splunk\System\_introspection\colddb\inflight-db_1517868285_1517867401_72' failed (reason='There are no more files.')
0 Karma
1 Solution

CooperInvestors
Engager

After much deliberation, the problem has been resolved.

It turned out to be a permissions issue on the Cold bucket storage.

The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.

To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.

View solution in original post

0 Karma

CooperInvestors
Engager

After much deliberation, the problem has been resolved.

It turned out to be a permissions issue on the Cold bucket storage.

The splunkd service on windows was running as local system. The Server$ computer account had read write access to the Cold bucket folders, and copying/moving/creating folders worked as expected when running as local system. For some reason this didn't work for splunk.

To resolve this, a domain service account was created and the permissions re-applied for this account. After changing the user splunkd ran as, the cold buckets started moving.

View solution in original post

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!