Moving indexes to a new Splunk server

[erga00]

I'm retiring an old Splunk server and replacing it with two new Splunk servers and I have some questions regarding the process to migrate the indexes.

From researching this site, I know that I need to:

1. Roll hot buckets to warm

2. Create identical indexes at target and copy folders from warm & cold folders on the source

3. Restart Splunk

Couple of questions:

1. Folder naming convention is db_START_END_ID. Do I need to change the IDs on the destination server? There's bound to a conflict at some point, how does Splunk handle it?

2. Any issues splitting up the folders between two target servers? For example, odd IDs go to target1 and even go to target2. Not required but would like to improve performance.

BTW I'm running Splunk 4.1.6 on Windows 2008 R2 x64. Source & target are the same Splunk & WIndows revs.

Thanks

[jbsplunk · Accepted solution]

To answer your questions:

1) If there is a conflict in bucket id's, splunk won't start, you would see an error about a conflict in your splunkd.log that looks like this:

ERROR IndexProcessor - caught exception for index=indexname during initialzation: 'Splunk has detected that a directory has been manually copied into its database, causing id conflicts [/opt/splunk/var/lib/splunk/indexname/db/db_epoch_epoch_1, /opt/splunk/var/lib/splunk/indexname/db/hot_v1_1].'.Disabling the index, please fix-up and run splunk enable index.

At that point, You'd need to check the .bucketManifest file in $SPLUNK_HOME/var/lib/splunk//db/ to see what the next bucket Id should be and rename the bucket so that the id does not conflict. So, if you already have a bucket called db_start_end_01 and you copy db_start_end_01 into the index, splunk won't start.

Checking the .bucketManifest file may show you that the next expected bucket id is 3, so you'd rename the folder you copied into the index to db_start_end_3. For instance, a splunk instance of mine has the following .bucketManifest in a particular index:

0 : db_1299201035_1297589013_0
1 : db_1193132257_1134909658_1
2 : db_1245221872_1244790735_2

If I had another bucket id ending in 0, I would need to rename it to db_start_end_3 or greater to avoid a conflict.

More information on this topic can be found here:

http://www.splunk.com/base/Documentation/latest/Admin/Restorearchiveddata

http://www.splunk.com/wiki/Community:MoveIndexes

2) With regard to splitting buckets, you shouldn't have any issues moving the data between the two servers so long as you keep the bucket id in mind. I wouldn't recommend this approach unless you are using a search head and the two target servers are both search peers.

[jbsplunk]

You are correct in that Splunk will not reuse the id's from transplanted buckets.

[vasanthmss]

check this, http://wiki.splunk.com/Community:MoveIndexes

[erga00]

The target servers don't have any data on them so I don't have to worry about ID conflicts with existing buckets. Reading between the lines of your reply, I gather that splunk will not reuse the IDs from transplanted buckets so I should be good to go.

I didn't mention it, but yes I will we using a search head and configure dist search on the target servers.

Thanks for the reply.