Advice for migrating to new hardware?

[oreoshake]

I'm in the process of migrating to new hardware for my indexers. The easiest way to do this would be:

1. Setup new indexer
2. Rsync var/lib/splunk (can I omit _internal or any other indexes?)
3. roll all indexes to warm
4. rsync again
5. Update DNS entries to point to the new host

There will be a small amount of data loss during this transition. I've also thought about converting my indexer to a forwarder, cutting the DNS entries over, and then rsyncing afterward (being very careful not to conflict with the new data).

Any success stories on migrating to new hardware?

[jrodman · Accepted solution]

Here's a rough document I wrote around this topic, but it's more about the configuration data and being sure it will work in the new environment, rather than the live handoff: http://www.splunk.com/wiki/Deploy:Migrating_a_Splunk_Install

When you rsync you might want to omit traversing the hot buckets, as you could end up doing a large amount of I/O for temporary files, ie. transferring the same data many times.

I recommend the hulahoop approach of parallel installs. It gives you much more flexibility to deal with problems and ensure correctness. Once you have your parallel install proven to work nicely, you can run clean eventdata on it, and begin bringing data over from your old system.

You can merge the indexes, so long as you are sure to cut it off around the time that you brought up the new indexer, and avoid id collisions. I wrote a script in an attempt to ease the merging of indexes (handle the id collisions), but it isn't publically facing yet. It's not a hard job, just renaming the directories so the third number doesn't collide.

[hulahoop]

Hi Neil, our general recommendation is to have Splunk on the new and old hardware operating in parallel during the migration. You can either split your data streams to both indexers or have the existing Splunk server index and forward to the other. Once you've verified the new indexer is running as expected, then you can make the switch to direct data only to the new indexer and retire the old one. This way data loss is less of a risk, and you have the chance to make changes to the new environment without affecting the existing production environment. This is the approach we've taken to migrate many production environments.

[oreoshake]

That sounds like a good strategy, I guess I would just sync everything except the hot buckets on the old server? Assuming no new hot buckets created after the manual roll happen to roll to warm during the transition period.