Deployment Architecture

ProcessTracker / fsck / bloomfilter - Failed: Failed to move bloomfilter to final dir:

lmyrefelt
Builder

HI,

After trying to move and separate the bloomfilter creation to a separate disk(s) and using a volume-stanza for bloomfilters as follows .. i also configured a longer range (365 days) to generate the bloomfilters for :
[default]
maxBloomBackfillBucketAge = 365d
[volume:bloom]
path = $SPLUNK_BLOOM/

$SPLUNK_BLOOM is configured under splunk-launch.conf ;
SPLUNK_BLOOM=/opt/splunk_db/cold_frozen2/bloom

And i can see it creates all the directories that it needs
/opt/splunk_db/cold_frozen2/bloom//bloomfilter/db_1423610749_1423588528_711/

02-11-2015 10:22:13.632 +0100 INFO ProcessTracker - (child_273__Fsck) Fsck - (bloomfilter only) Rebuild for bucket='/opt/splunk_db/hot_warm//db/db_1413936005_1413636034_85' took 31.29 seconds

02-11-2015 10:22:13.632 +0100 WARN ProcessTracker - (child_273__Fsck) Fsck - Write of bloomfilter to='/opt/splunk_db/cold_frozen2/bloom/index-name/bloomfilter/db_1413936005_1413636034_85' failed: Failed to move bloomfilter to final dir: Cannot rename src="/opt/splunk_db/hot_warm/index-name/db/db_1413936005_1413636034_85-tmp/bloomfilter.tmp" to dst="/opt/splunk_db/cold_frozen2/bloom/index-name/bloomfilter/db_1413936005_1413636034_85/bloomfilter": Invalid cross-device link

After / On a restart i get the message that fsck started to rebuild bloomfilters ... however the path they these messages are pointing to is the old path / standard path to "hot / Warm" .

I belive

Invalid cross-device link

might be a key here ?

ANyone else that have been moving around on the bloomfilter structure ?

Edit: i found this one that is also related to the issue;
02-11-2015 10:53:26.250 +0100 WARN BucketMover - BucketManifestUpdateExitHandler: process handling bucket="db_1413936005_1413636034_85" exited with code=102; search for any previous messages that might have been produced by the external process

0 Karma
1 Solution

lmyrefelt
Builder

The problem i have is that it does not seem to generate / being able to move the temporary bloom filters its seems to create to its final a new directory on separate disks specified / configured as volumes inside splunk as per described in indexes.conf.spec .

Support case / Enchantment request:
224317

View solution in original post

0 Karma

lmyrefelt
Builder

The problem i have is that it does not seem to generate / being able to move the temporary bloom filters its seems to create to its final a new directory on separate disks specified / configured as volumes inside splunk as per described in indexes.conf.spec .

Support case / Enchantment request:
224317

0 Karma

lmyrefelt
Builder

This has been "accepted" as an bug and should be scheduled to be fixed, according to support it should be pretty easy.
:)

Closing it.

0 Karma

mikaelbje
Motivator

Recommendation from case 224317: This issue was fixed by SPL-99961 after review by Dev Team on your issue. Fix Version/s: 6.2.4

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...