How do I migrate a copy of Splunk Enterprise server to a new machine?

[mike_k]

I would like to take a copy of my Production standalone Splunk instance and stand it up as a development machine.

My Production machine is running on Linux and I'd like to move a copy to a new Linux server (different hostname, domain).

Since i don't want to move the data stored in the indexes, I was wondering whether i can just copy the contents of the $SPLUNK_HOME/etc folder? or are there further files that need copying across (e.g kvstore settings)?

... or do i really need to copy the whole contents of $SPLUNK_HOME and then delete the index data from the development machine after the copy has finished?

[chaker]

Is it correct to say:
- that changing the GUID means any datamodel acceleration's will be duplicated?
- changing the splunk secret means none of the hashed values in the ect backup can be read?

[isoutamo]

I cannot answer for this first 100% sure, but I don’t think that GUID is in use on DMA. 

2nd one is true, if you are just copied those from one env to another. But in security point of view it’s not a good idea to do a direct copy from prod to test.

[chaker]

Yes!

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Backupconfigurations

https://lantern.splunk.com/Splunk_Success_Framework/Platform_Management/Managing_backup_and_restore_...

$SPLUNK_HOME/etc/ includes all config to stand up in another environment.

Treat the KVStore like data, and backup it up and restore in addition to the $SPLUNK_HOME/etc/

[isoutamo]

Hi

There is some things which you must check and modify when you are doing your lab env from production. See e.g. https://lantern.splunk.com/Splunk_Success_Framework/Platform_Management/Setting_up_a_lab_environment

Personally I'm not prefer to just copy production conf to the new lab host as there are at least the next files which must remove / change.

• License, you cannot use a same production license on test that you are using in prod. You should ad your lab as license client or better is to get developer or dev/test license from splunk (https://dev.splunk.com/enterprise/dev_license/)
• GUID ($SPLUNK_HOME/etc/instance.cfg)
• splunk.secret ($SPLUNK_HOME/etc/auth/splunk.secret) used as a seed from crypting passwords etc.
• Host name, TLS settings etc.
• indexes (volume etc.) configurations etc.

Usually it's much easier just install splunk from scratch and then apply those apps & another configurations from git or other version control system. When you have several environments / nodes it's almost mandatory to use e.g. git where to store all configurations and don't use GUI for changing environments.

If you haven't for storing apps etc. then you should start to use it. Maybe the easiest way to get those from your production is something like 

for i in $(splunk search "| rest /services/apps/local f=core f=title f=disabled|where core=0 AND disabled=0|table title" |tail +3); do splunk package app $i;done

Then copy those spl files from $SPLUNK_HOME/share/splunk/app_packages to the new node and just install those by "splunk install app <app>.spl".

r. Ismo

[mike_k]

@isoutamo, thanks for that feedback.

Yes i agree there are a number of changes that need to be done after copying across the content of the etc directory and it can be fiddly.

I have covered off those points that you mentioned. I had forgotten about the GUID (thanks for that). When you mention changing "indexes (volumes etc)" do you mean here that just need to confirm that the sizes allocated to the various indexes matches the storage available in the test environment (which i still need to do).  I've also corrected:

• local passwords (using user-seed.conf)
• LDAP settings in authentication.conf.
• Deployment app changes (so Forwarders point to new test Indexer IP address)

From what i understand the primary location of hostname is just in server.conf isn't it or is it located in other places as well?

I do like the concept of having an install from scratch and then exporting individual apps from the Production environment into my test environment.

What do you do for system configuration files that aren't part of an app? (e.g SPLUNK_HOME/etc/system). Do you just copy these files off individually and maintain them individually in GIT? Can you also package deployment apps in this same fashion?