I'm retiring an old Splunk server and replacing it with two new Splunk servers and I have some questions regarding the process to migrate the indexes.
From researching this site, I know that I need to:
1. Roll hot buckets to warm
2. Create identical indexes at target and copy folders from warm & cold folders on the source
3. Restart Splunk
Couple of questions:
1. Folder naming convention is dbSTARTEND_ID. Do I need to change the IDs on the destination server? There's bound to a conflict at some point, how does Splunk handle it?
2. Any issues splitting up the folders between two target servers? For example, odd IDs go to target1 and even go to target2. Not required but would like to improve performance.
BTW I'm running Splunk 4.1.6 on Windows 2008 R2 x64. Source & target are the same Splunk & WIndows revs.
To answer your questions:
1) If there is a conflict in bucket id's, splunk won't start, you would see an error about a conflict in your splunkd.log that looks like this:
ERROR IndexProcessor - caught exception for index=indexname during initialzation: 'Splunk has detected that a directory has been manually copied into its database, causing id conflicts [/opt/splunk/var/lib/splunk/indexname/db/db_epoch_epoch_1, /opt/splunk/var/lib/splunk/indexname/db/hot_v1_1].'.Disabling the index, please fix-up and run splunk enable index.
At that point, You'd need to check the .bucketManifest file in $SPLUNKHOME/var/lib/splunk/
Checking the .bucketManifest file may show you that the next expected bucket id is 3, so you'd rename the folder you copied into the index to dbstartend_3. For instance, a splunk instance of mine has the following .bucketManifest in a particular index:
0 : db_1299201035_1297589013_0 1 : db_1193132257_1134909658_1 2 : db_1245221872_1244790735_2
If I had another bucket id ending in 0, I would need to rename it to dbstartend_3 or greater to avoid a conflict.
More information on this topic can be found here:
2) With regard to splitting buckets, you shouldn't have any issues moving the data between the two servers so long as you keep the bucket id in mind. I wouldn't recommend this approach unless you are using a search head and the two target servers are both search peers.
The target servers don't have any data on them so I don't have to worry about ID conflicts with existing buckets. Reading between the lines of your reply, I gather that splunk will not reuse the IDs from transplanted buckets so I should be good to go.
I didn't mention it, but yes I will we using a search head and configure dist search on the target servers.
Thanks for the reply.