Installation

How to migrate Splunk enterprise to new hardware

vnguyen46
Contributor

Is anyone here can share the best practice on how to migrate Splunk enterprise to new hardware? my system include:
2 deployment servers (one for each zone)
4 HFs (two for each zone)
1 cluster master
7 indexers
3 SHs

I am advised to copy entire /etc directory from old to new systems (except indexers), but I am wondering which files I need to reconfigure as servers come with new hostnames and IPs.

Thanks,

Labels (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi vnguyen46,
on Linux if you can use the same hostnames and IP addresses of old servers it's very easy (and advisable) you can copy the entire /opt/splunk directory and your installation will run.
If you cannot use the same hostnames and IPs, you can do the same thing, but you have to manually modify many parameters that I suggest to find using grep, anyway they are:

  • hostname in $SPLUNK_HOME/etc/system/local/server.conf
  • hostname in $SPLUNK_HOME/etc/system/local/inputs.conf.conf
  • all the addresses of the cluster and search heads,
  • repeat the commands for automatic start of Splunk.

If instead you have Windows servers, you have to install Splunk on each server and then copy the $SPLUNK_HOME/etc folder, then do the same modification of linux

Anyway, follow the indication in Splunk docs for each of your environments: Indexers cluster, Search Heads, Deployment Servers.

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vnguyen46,
on Linux if you can use the same hostnames and IP addresses of old servers it's very easy (and advisable) you can copy the entire /opt/splunk directory and your installation will run.
If you cannot use the same hostnames and IPs, you can do the same thing, but you have to manually modify many parameters that I suggest to find using grep, anyway they are:

  • hostname in $SPLUNK_HOME/etc/system/local/server.conf
  • hostname in $SPLUNK_HOME/etc/system/local/inputs.conf.conf
  • all the addresses of the cluster and search heads,
  • repeat the commands for automatic start of Splunk.

If instead you have Windows servers, you have to install Splunk on each server and then copy the $SPLUNK_HOME/etc folder, then do the same modification of linux

Anyway, follow the indication in Splunk docs for each of your environments: Indexers cluster, Search Heads, Deployment Servers.

Bye.
Giuseppe

0 Karma

vnguyen46
Contributor

Giuseppe - really appreciate your help. There is another idea I'd like to ask if it's practical.
Retaining the DS and Cluster Master running on VM Linux. Adding new bare-metal based indexers, SHs and HFs to the cluster. Removing the old instances from the system.

Do you think it sounds doable?

Thanks,
Vincent

0 Karma

vnguyen46
Contributor

Giuseppe - Thank you so much for all your responses.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vnguyen46,
Indexers have an high access to I/O so they need to have high performance disks (at least 800 IOPS), so physical servers are a good idea, but only if they have at least SAS 15k disks or SSH.

If you haven't all 15k disks you can configure your indexes to put hot and warm buckets on SAS 15k disks and and cold buckets on other slower disks.

DS, CM and SHs don't need to be installed on Physical servers and can be on virtual appliaces because they don't have an high use of disks.

When you choose hardware, put attention to the number of CPUs of your servers because Splunk requests are, for Indexers, at least 12 CPUs and 12GB of RAM and, for SHs, at least 16 CPUs and 12 GB of RAM.

For more details see https://docs.splunk.com/Documentation/Splunk/7.3.2/Capacity/Referencehardware .

Bye.
Giuseppe

0 Karma

vnguyen46
Contributor

Thanks - the new servers exceed all that numbers. For the migration, don't you think I can add new servers IDX, HF, SH to the corresponding clusters, then remove the old servers?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vnguyen46,
Yes for Indexers Cluster, for the others, you should manually configure one by one.

Put much attention to the Master Node and follow the documentation, eentually if you could use the same hostname and IP address it's very useful!

Put attention to Deployment Servers: I hope that you have the deploymentclient.conf file in a dedicated TA.
If not I suggest to use this situation to create a dedicated TA for each of your zones containing the address of the related Deployment Server so you can in future manage it by DS, the problem is to manully remove the old one in $SPLUNK_HOME/etc/system/local.
If instead you already manage the deploymentclient.conf in a TA, you have to configure the new DS and then deploy the new TA that addresses th new DS.

About SHs, you didn't mentioned a cluster so I think that they are stand alone, so you can install the new SHs and manually configure them one by one.

Bye.
Giuseppe

P.S.: if you're satisfied by this answer, please accept and/or upvote it.
See next time.

0 Karma

vnguyen46
Contributor

It's all a great idea. That's surely helpful. Thank you,

0 Karma

vnguyen46
Contributor

Hi Giuseppe,
You've a great knowledge on the migration and I'd like to follow up for help.
"If instead you have Windows servers, you have to install Splunk on each server and then copy the $SPLUNK_HOME/etc folder, then do the same modification of linux"
1. Does copying /etc folder work for Linux base as well? All my servers are running on Linux.
2. The existing system is running on Splunk Enterprise 7.0. Is there any version conflict if I copy this /etc folder to the new system running v7.3?

3. I have Deployment server and Cluster Master in one VM box. It's causing some issues and hard to troubleshot. Any advises on splitting the roles onto two separate servers, one VM for each role?
Thanks,

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vnguyen46,
I start from issue 3 because is the most relevant: as you can see at https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Systemrequirements , it isn't a good idea to have on the same server Deployment Server and Master Node:

  • Deployment Server must be on a dedicated server if it manage more than 50 clients, if less could be shared with other roles but not Master Node;
  • Master Node could be shared with less heavy servers as License Master or Deployer, never with Deployment Server, especially if you have many clients!

About the conflict versions I don't know any one and I migrated from 4 to 5, from 5 to 6 and from 6 to 7 without problems, anyway the best approach is:

  • to install the same version you have,
  • copy etc folder.
  • then upgrade version.

in this way you're sure that you haven't any upgrade problem.

About the first issue, it's not relevant because you haven't Windows (good idea!).

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...