Question on migrating a small Splunk Enterprise environment to new hardware

[danielsheerin]

Hi all,

We have a relatively small Splunk environment that has 2 Universal forwarders and 1 Indexer on separate servers in the same domain. Our hardware is reaching end of life and as a result we have to migrate to new hardware. This is an environment we only recently took ownership for and the knowledge is not where it should be for a project such as this. As such, I wanted to ask here for some pointers on how to carry out such a project.

We have our new servers. Old servers were Windows Server 2008 R2 and new servers are 2012 R2. We use a service account that has access to all of our servers to run the forwarders rather than having light forwarders on every server. It is an option to not migrate our old buckets, etc. as we mainly use Splunk for live monitoring rather than historical tracking, etc.

Would I be correct in assuming that the ideal (high level) plan would be to backup everything, upgrade the existing Indexer to the latest version of Splunk, copy over the $Splunk home directory to the new server, install Splunk on the new server, check the conf files to make sure the new servers IP, etc is updated where needed and then start Splunk on the new server? We will also be migrating one of our forwarders. Is it possible to do this after upgrading the Indexer server?

Also, in regards to backing out if things go wrong is it as straightforward as turning of Splunk on the new servers and pointing the forwarders back to the old server? If I have forgotten anything please let me know. I am very new to this and am willing to listen to all the advice I can get. Thank you in advance for any replies.

[woodcock]

We just did this and here is the approach.

1: Clone the other indexer and rsync the /opt/splunk/var/lib/splunk/ area once more when you are done (will take a LONG time). This can be done while the old indexer is still the live/main one and while it is running. Keep running periodic rsync until right before the cutover window.
======Start of MW/cutover=====
2: On the new indexer, edit every /opt/splunk/var/lib/splunk/*.dat file and add 500 to it.
3: Start Splunk on the new indexer.
4: Point a forwarder to the new indexer.
5: Make sure all is good.
6: Point all fowarders to the new indexer.
7: Wait until all forwarders are forwarding in to the new indexer and make sure that none are forwarding in to the old one.
8: Stop Splunk on the old forwarder.
9: Do one last rsync but be sure to EXCLUDE the /opt/splunk/var/lib/splunk/*.dat files. THIS EXCLUSION PART IS VERY IMPORTANT!!!
These files tell Splunk what number to use to create the next hot bucket. We bumped this up by 500 so that the new buckets are far apart from the old buckets so that we can have both indexer running at the same time and we will not have overlapping filenames.
This allows us to copy the old buckets while the new indexer is running and be sure that we will not be overwriting any files or directories that the new indexer is writing.

[danielsheerin]

Thank you for your answer. I will be attempting this migration tomorrow or Thursday depending on workload. It looks as if we will be installing the same version that we are running currently and trying to get both up and running before we upgrade the new Splunk environment.

[esix_splunk]

Good Question. There are a few approaches here, and Ill break this down by Indexer Upgrade / migration and Forwarder Upgrade / Migration.

First the Indexer Upgrades. I'd first upgrade the indexers. Makes life easier. Then for migrating, there are a few options here. Depending on the volume of data you have existing, you can migrate this data by copying over the $splunk/var/lib/ structure (This can be tb+ worth of data depending how long you've been running.)

Alternatively, if you dont have an immediate requirement to migrate the data, or keep it long term, why not run these in parallel. Keep the existing indexer up and running with the historic data and use the new indexer to collect new data. Your search head can run distributed search across both indexers. Easy.

For your forwarders, upgrade them at your convenience. Use a deployment server (if youre not currently) to manage your outputs. You can update your forwarder app, which will point it to the new indexer. Or ideally, you can just change DNS for the old indexer and point it at the new indexer. Then you dont need to change configurations on the forwarders, just run your upgrade plays.

Cheers
Eric

[danielsheerin]

Thank you for the reply, Eric. My director is concerned about upgrading and migrating on the same night. They are wondering if it is possible to install Splunk Enterprise 6.4.1 on the new 2012 server and make sure both are operational after migrating the core components. We would then upgrade to version 7.0.1 after a period and we're happy things are functioning as expected. We use Splunk mainly for real time monitoring for production log files and as such rely on it quite heavily.

[esix_splunk]

This method is works also. 🙂