Hi,
I recently had to move my hot/warm buckets for my splunk indexes to a new linux device on the same machine.
I use auto scaling on all my buckets
I used rsync -azv to copy over the hot/warm buckets from one directory to another so it looked like this:
rsync -azv /var/lib/splunk/index1/db /var2/lib/splunk/index1/
i made sure to change the ownership with chown -R splunk for the directory
I turned off splunk
i ran rsync on all the folders for the indexes i was moving
updated splunk-launch.conf to change $SPLUNKDB from /var/lib/splunk to /var2/lib/splunk
then i updated the index.conf files for etc/apps/search/local and for etc/system/local to $SPLUNK/index* (for each indexes)
When i started splunk up it immediately disabled all the indexes except _internal and one of the indexes that i had moved first as a test. The only way i could get splunk to work is to edit the indexes.conf file and have it make the home buckets in a new directory.
However the indexes in this new directory now have new hot buckets and now i can only search for events after the switch.
How do i reconcile these buckets. I want to get all the data from before the switch to be in the same folder. How do i do this without causing bucket naming collisions?
It's a classic mistake, when using rsync, you have to exclude the hot buckets, to avoid duplicates bucket id when they rotate to warm.
see those posts :
http://splunk-base.splunk.com/answers/30986/why-is-my-index-disabled
http://splunk-base.splunk.com/answers/6114/whats-this-duplicate-bucket-in-my-index
To resume, identify the duplicates (one hot_X_
It's a classic mistake, when using rsync, you have to exclude the hot buckets, to avoid duplicates bucket id when they rotate to warm.
see those posts :
http://splunk-base.splunk.com/answers/30986/why-is-my-index-disabled
http://splunk-base.splunk.com/answers/6114/whats-this-duplicate-bucket-in-my-index
To resume, identify the duplicates (one hot_X_
This worked great, but duplicate warm buckets caused a conflict as well
Yep, you need to have a unique id for each bucket. That is the last set of digits. You can manually rename the directory names, as long as you an unused number. Don't mess with the epoch timestamps, though.
/K
you have a conflict with buckets 13 and 567 according to the errors. just change that value - not the 1366xxxxx parts.
See Yann's posts as well.
Thanks for your response: how do i not mess with the epoch time stamps
04-16-2013 19:33:27.165 -0700 ERROR DatabaseDirectoryManager - idx=pf_app_mobile bucket=db_1366089008_1365567758_13 Detected directory manually copied into its database, causing id conflicts [path1='/var2/splunk/lib/splunk/pf_app_mobile/db/hot_v1_13'
-0700 ERROR IndexProcessor - caught exception for idx=pf_systems during initialization: 'idx=pf_systems bucket=hot_v1_567 Detected directory manually copied into its database, causing id conflicts [path1='/var2/splunk/lib/splunk/pf_systems/db/db_1366114180_1366085206_567' path2='/var2/splunk/lib/splunk/pf_systems/db/hot_v1_567'].'
what did the error message say? STDERR and in splunkd.log