Splunk Search

Bucketmover - aborting move because recursive copy from src to dest failed (no such file or directory)

banderson7
Communicator

Getting low on warm space for my buckets, so I changed the maxHotSpanSecs to 6100000 or ~70 days. After restarting the indexer, I'm getting the error messages in the title. Can someone assist w/ this?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Did you restart Splunk as the wrong user? Typically only applies to linux distros.

If so you'll need to stop Splunk and then recursively change the owner to fix file permission issues:

 chown -Rf correctUser:correctGroup /path/to/splunk

Then switch to the correct user and start Splunk.

0 Karma

banderson7
Communicator

It's running as the right user, and there are new buckets from the indexes (with the correct permissions), but I'm still getting the error message. I was also expecting more buckets in the cold directories/partition, after changing the setting and restarting.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Hot buckets (typically only 3) roll to warm buckets (typically 200+) before they role to cold.

0 Karma

banderson7
Communicator

Does the fact that I changed the maxHotSpanSecs speed up the rolling process (instead of 1 roll of three buckets per day, more buckets roll per day)? If not, how do I do that? I'd like to clear up some space on my warm partition of data we don't search that often.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Bucketing and retirement policies are tricky. We kinda got off subject though. I have to read the docs every time I mess with indexes.

http://docs.splunk.com/Documentation/Splunk/6.0.2/Indexer/Setaretirementandarchivingpolicy

0 Karma

jkat54
SplunkTrust
SplunkTrust

The bucket roll will occur on a restart but it can take some time. It also needs to be triggered by the settings. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes

In short... All events in the bucket need to match the condition prior to the roll.

There is also a command that will force bucket rolls.

0 Karma

jkat54
SplunkTrust
SplunkTrust

To speed up the roll, given the fact that all events must match the conditions, we typically reduce the max data size and frozen time period:

 The maxTotalDataSizeMB and frozenTimePeriodInSecs attributes in indexes.conf help determine when buckets roll from cold to frozen.
0 Karma

banderson7
Communicator

I'm looking to roll from warm to cold, so I'd set maxWarmDBCount to less than 300 I'd imagine. Don't see why I changed maxHotSpanSecs now that I think about it. Does that sound right for what I'm trying to do?

Also, can you think of why I'm getting the error regarding recursive copy?

Thanks very much for your help, btw.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Agreed, the maxHotSpanSecs isn't as much help as the maxTotalDataSizeMB and maxDataSize. Sorry i was on mobile, but now home... this is the end all resource you need to thoroughly read & understand: http://docs.splunk.com/Documentation/Splunk/6.0.1/admin/Indexesconf

The recursive copy error was most likely due to file permissions. Perhaps splunk ran as root once upon a time... made some hot/warm buckets owned by root, and now your new setting is trying to move those buckets. You should be able to find more issues "around/near" that error message if you look in index=_internal component=bucketmover or just index=_internal. Its telling you it cant find the file to copy, or cant find the destination to copy to. So either coldDBPath / WarmDBPath is not available... or you dont have permission to access those paths, or you dont have permission to access the bucket its trying to roll (by you I mean the splunkd user).

0 Karma

banderson7
Communicator
03-14-2016 14:26:59.302 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/application/db/db_1354776505_1354743198_475' to dst='/mnt/splunk/cold/appl
ication/colddb/inflight-db_1354776505_1354743198_475' failed (reason='No such file or directory')
03-14-2016 14:26:59.333 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/security/db/db_1354759131_1354748274_3081' to dst='/mnt/splunk/cold/securi
ty/colddb/inflight-db_1354759131_1354748274_3081' failed (reason='Permission denied')
03-14-2016 14:26:59.999 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/application/db/db_1354743197_1354664613_474' to dst='/mnt/splunk/cold/appl
ication/colddb/inflight-db_1354743197_1354664613_474' failed (reason='No such file or directory')


/mnt/splunk/cold/application/colddb/inflight-db_1354743197_1354664613_474:
total 304624
drwx--x--x   3 splunk splunk      4096 2016-03-14 14:29 .
drwx------ 721 splunk splunk     53248 2016-03-14 14:28 ..
-rw-------   1 splunk splunk  17888225 2016-03-14 14:29 1354742336-1354738684-6468197426103594966.tsidx
-rw-------   1 splunk splunk 293102760 2016-03-14 14:28 1354743197-1354664613-2556498674997796822.tsidx

/mnt/splunk/cold/application/colddb/inflight-db_1354776505_1354743198_475:
total 86908
drwx--x--x   3 splunk splunk     4096 2016-03-14 14:28 .
drwx------ 721 splunk splunk    53248 2016-03-14 14:28 ..
-rw-------   1 splunk splunk 77457453 2016-03-14 14:28 1354773028-1354743198-1990108998233234267.tsidx
-rw-------   1 splunk splunk 11048518 2016-03-14 14:28 1354775364-1354753985-1692011677748308117.tsidx

I think we get an error, and then the copy continues anyway. That's what this looks like anyway. Also, do I want to adjust maxTotalDataSizeMB, isn't that for all the data in the index?maxDataSize is for the hot buckets I think. Should I edit maxWarmDBCount to roll warm to cold? That says to roll at a certain number, but there's less control over the amount of time the data spends in warm buckets, right?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yep it's a permission denied error. You should try the recursive chown. Bucket mover errors can quickly stack up. maxDataSize and the number of buckets hot, and warm are the ones you're most interested in.

0 Karma

jkat54
SplunkTrust
SplunkTrust

As well as frozenTimePeriodInSecs if you're wanting to roll to frozen/delete

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...