Is anyone here can share the best practice on how to migrate Splunk enterprise to new hardware? my system include:
2 deployment servers (one for each zone)
4 HFs (two for each zone)
1 cluster master
I am advised to copy entire /etc directory from old to new systems (except indexers), but I am wondering which files I need to reconfigure as servers come with new hostnames and IPs.
on Linux if you can use the same hostnames and IP addresses of old servers it's very easy (and advisable) you can copy the entire /opt/splunk directory and your installation will run.
If you cannot use the same hostnames and IPs, you can do the same thing, but you have to manually modify many parameters that I suggest to find using grep, anyway they are:
If instead you have Windows servers, you have to install Splunk on each server and then copy the $SPLUNK_HOME/etc folder, then do the same modification of linux
Anyway, follow the indication in Splunk docs for each of your environments: Indexers cluster, Search Heads, Deployment Servers.
Giuseppe - really appreciate your help. There is another idea I'd like to ask if it's practical.
Retaining the DS and Cluster Master running on VM Linux. Adding new bare-metal based indexers, SHs and HFs to the cluster. Removing the old instances from the system.
Do you think it sounds doable?
Indexers have an high access to I/O so they need to have high performance disks (at least 800 IOPS), so physical servers are a good idea, but only if they have at least SAS 15k disks or SSH.
If you haven't all 15k disks you can configure your indexes to put hot and warm buckets on SAS 15k disks and and cold buckets on other slower disks.
DS, CM and SHs don't need to be installed on Physical servers and can be on virtual appliaces because they don't have an high use of disks.
When you choose hardware, put attention to the number of CPUs of your servers because Splunk requests are, for Indexers, at least 12 CPUs and 12GB of RAM and, for SHs, at least 16 CPUs and 12 GB of RAM.
For more details see https://docs.splunk.com/Documentation/Splunk/7.3.2/Capacity/Referencehardware .
Thanks - the new servers exceed all that numbers. For the migration, don't you think I can add new servers IDX, HF, SH to the corresponding clusters, then remove the old servers?
Yes for Indexers Cluster, for the others, you should manually configure one by one.
Put much attention to the Master Node and follow the documentation, eentually if you could use the same hostname and IP address it's very useful!
Put attention to Deployment Servers: I hope that you have the deploymentclient.conf file in a dedicated TA.
If not I suggest to use this situation to create a dedicated TA for each of your zones containing the address of the related Deployment Server so you can in future manage it by DS, the problem is to manully remove the old one in $SPLUNK_HOME/etc/system/local.
If instead you already manage the deploymentclient.conf in a TA, you have to configure the new DS and then deploy the new TA that addresses th new DS.
About SHs, you didn't mentioned a cluster so I think that they are stand alone, so you can install the new SHs and manually configure them one by one.
P.S.: if you're satisfied by this answer, please accept and/or upvote it.
See next time.
You've a great knowledge on the migration and I'd like to follow up for help.
"If instead you have Windows servers, you have to install Splunk on each server and then copy the $SPLUNK_HOME/etc folder, then do the same modification of linux"
1. Does copying /etc folder work for Linux base as well? All my servers are running on Linux.
2. The existing system is running on Splunk Enterprise 7.0. Is there any version conflict if I copy this /etc folder to the new system running v7.3?
3. I have Deployment server and Cluster Master in one VM box. It's causing some issues and hard to troubleshot. Any advises on splitting the roles onto two separate servers, one VM for each role?
I start from issue 3 because is the most relevant: as you can see at https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Systemrequirements , it isn't a good idea to have on the same server Deployment Server and Master Node:
About the conflict versions I don't know any one and I migrated from 4 to 5, from 5 to 6 and from 6 to 7 without problems, anyway the best approach is:
in this way you're sure that you haven't any upgrade problem.
About the first issue, it's not relevant because you haven't Windows (good idea!).