Merging v4 and v5 indexes

[Lucas_K]

I have previously merged multiple v4 indexes together with no issues (ie. http://wiki.splunk.com/Community:MoveIndexes - Last updated 18th April 2013.).

My question is, are there any v5 specific issues or updates tasks required? Note: Our v5 indexers are not clustered at all yet so bucket renumbering tasks remains the same as with a v4 merge.

I noticed that v5 indexes have a "indexname.dat" file that simply contains the next hotbucket number. Does this need to be deleted or manually updated, in a similar fashion to how the manifests should be deleted, so that upon splunk start up they recreated using the combined buckets?

[Lucas_K]

That still didn't answer my question if its really required to delete the .dat file however.

I guessed that if deleted splunk would recreate that based on the buckets seen (just like the bucketmanifest). I wasn't sure it if needed to be deleted first before it would update it when another bucket is dropped in however (ie. waht would splunk do if .bucketManifest still existed would it update this or ignore the new buckets).

I actually ended up using the .dat file as information for my bucket renaming perl script. It actually makes it easier than v4 merges as the .dat number takes into account cold bucket numbers also (something I missed on my first merge as our cold buckets were on a different volume!).

Use that number, add a little to it (just so I can see the difference between old and foreign) then rename all foreign buckets starting at that number. Then rename the local hot bucket to a number after that. Delete the .bucketManifest and .dat file. Then restart. All foreign buckets are seen and then rolled into cold 🙂

We also ran into a weird issue of "license duplication" where a currently running instance saw foreign buckets dropped into a deeper directory within cold volume (ie. running cold index = /volumes/cold/my_index/db, foreign import dir = /volumes/cold/IMPORT/my_index/db). As the cold volume was the only partition large enough to hold the initial copy I had no choice but to copy it in there (/volumes/cold was also the top dir of the mount).

The question about this I have is why did splunk see these indexes? indexes.conf didn't have a reference to the imported dir so why did splunk look inside it? My work around was just to chmod 700 the dir from another account so splunk couldn't look inside it. We also couldn't afford the time to shutdown this particular index as there was a significant amount of foreign buckets data.

[yannK]

You can merge buckets from v4 in a v5 index. Just copy the bucket folder and restart splunk to detect them. Splunk Will recreate the missing bits.

if your index get's disable, check for bucket id duplicates : http://splunk-base.splunk.com/answers/30986/why-is-my-index-disabled