How to migrate Splunk Enterprise from one RDP to another?

[VenkataAnuradha]

Hi everyone,

To use the new Windows server2019–2022, which is OS compatible, we have planned to migrate the Search Head, Indexers, and Deployment Server instances of Splunk (old and new).

Since this is my first action, I need specific "backup and installation" instructions in order to complete the task safely.

Current Splunk install: Windows 2016 -  Splunk Ent 9.0.1
New Splunk install: Windows 2019/2022 running Splunk Ent 9.0.2

The architecture includes:

1 Search Head (OS - Windows 2016)
2 Indexers (OS - Windows 2016)
1 Deployement Server (OS - Windows 2016)
62 Universal Forwarders (OS - Linux)

Thanks in advance for any help.

[VenkataAnuradha]

Hi @gcusello 

Could you clear the doubts on the below points. 

• copy the $SPLUNK_HOME\etc folder on the new machines,  
  Should I copy the $SPLUNK_HOME\etc folder throughout all servers? ( Search Head, Indexers, Deployment Server). and is the single etc folder backup sufficient?
  

Now, it's relevant if you configured deploymentclient.conf and outputs.conf on your forwarders in a dedicated App, managed by Deployment Server, or they are in $SPLUNK_HOME/etc/system/local.

Sorry, I didn't get this point. Could you please explain in more detail

And with version 9.0.1, we have 62 universal forwarders. Do we need to make any changes to these UFs before or after installing Splunk Enterprise 9.0.2 on the SearchHead, Indexers, and Deployment Servers?

[gcusello]

Hi @VenkataAnuradha,

1)

no you have to copy each server in the new one: SH1 -> SH1new, etc...

then you have rto check configurations because I suppose that IP addresses of the new servers are different.

2)

you have to change deployment server addressing that's managed in a file called "deploymentclient.conf".

If you have it in $SPLUNK_HOME\system\local you cannot manage it by Deployment Server.

If you have it in a dedicated app, you have only to change the Deployment Server addressing on the Deployment  server which deploy the new addressing to all the UFs.

So it's a best practice to put this file in a dedicated app (together with outputs.conf)  instead in $SPLUNK_HOME\system\local.

Ciao.

Giuseppe

[gcusello]

HI @VenkataAnuradha,

you spoke of migrating your Splunk infrastructure to new servers, copying the $SPLUNK_HOME\etc folders, each machine to the corresponding new one, you already have all the configurations active.

About second question:

each Universal Forwarder is connected to a deployment server to deploy apps; this connection is managed in a conf file called deploymentclient.conf; this conf file can be created in two ways manually, editing file, or using the CLI command "splunk deploy-poll deploymentserver:8089".

If you created using the above command the file is located in $SPLUNK_HOME|etc\system\local, but in this way isn't manageable by Deployment Server.

So if it's in the above folder, you have to create a new deploymentclient.conf (addressing the new Deployment Server) and delete the old one.

If instead it's already in a dedicated add-on, you have only to modify the deploymentclient.cong.

The same thing for outputs.conf (addressing Indexers).

Ciao.

Giuseppe

[gcusello]

Hi @VenkataAnuradha,

proceding with order, you have to:

• install the new machines using the same Splunk Version (9.0.1);
• stopping the old machines,
• copy the $SPLUNK_HOME\etc folder on the new machines,
• only on Indexers, copy the folder where data are stored (by default $SPLUNK_HOME\var\lib\splunk) from the old Indexers to the new ones,
• run the new machines,
• update Splunk to the last version on all machines.

Now, it's relevant if you configured deploymentclient.conf and outputs.conf on your forwarders in a dedicated App, managed by Deployment Server, or they are in $SPLUNK_HOME/etc/system/local.

In the second case, you have to:

• create a new add-on, called e.g. TA_Forwarders, containing at least three files:
  • app.conf, containing a description of the app,
  • deploymentclient.conf, containing the address of the new Deployment Server,
  • outputs.conf, containing the addresses or the new Indexers,
• manually copy the TA in each Forwarder,
• manually delete old deploymentclient.conf and outputs.conf in the $SPLUNK_HOME/etc/system/local folder,
• restart Splunk on each Forwarder,
• at the same time, you could also upgrade the Forwarder versions.

If instead you already are managing Forwarders using a dedicated TA, you have to modify the distributed add-on changing the addresses of Deployment Servers and Indexers and then deploy the new add-on to the Clients.

Only for speaking: in many years I'm working on Splunk, I never saw any production infrastructure based on Windows, only test or labs or very very little infrastructucres, think to this

Ciao.

Giuseppe

[VenkataAnuradha]

Hi @gcusello 

Greetings of the day.

I would like to inform you that the downtime for this activity is consuming a lot of time which is an issue for us. Could you please clarify on the below points.

1) Can we move the backup from the old server to the new server in the pre-installation and then start the new Splunk version installation.

2) Is it possible to make changes to all 62 universal forwarders in a single shot prior to installing the latest Splunk version? Instead of copying and pasting the add-on into each universal forwarder. 

If yes, Could you please let me know the process.

[gcusello]

Hi @VenkataAnuradha,

Answering to your questions:

1)

if you have Linux it should be possible, but you have Windows, so you have to install, the same version of Splunk and then copy configuration and data,

this is another reason (not the main!) to prefer Linux to Windows!

but anyway it's the same thing, because you can install Splunk in the new servers when the old systems are still running and then after copy configuration and data, so you'll have the same downtime.

2)

you must upgrade UFs after the Splunk Central infrastructure is updated because you must have on UFs the same or lower version of Splunk, not greater.

Then you can upgrade all UFs also in one shot.

You didn't anwered to the main question about this topic: have you deploymentclient.conf in $SPLUNK_HOME\etc\system\local or in a dedicated app?

this is really important for the way (and the time) to change the new Deployment Server addressing.

Ciao.

Giuseppe

[PickleRick]

Actually, you can have newer UFs than your indexers. The only problem I've observed so far is that UFs 9.0 want to send their config changes to _configtracker index which is not configured by default on earlier indexer versions.

[VenkataAnuradha]

Hi @gcusello ,

2) you must upgrade UFs after the Splunk Central infrastructure is updated because you must have on UFs the same or lower version of Splunk, not greater.

Then you can upgrade all UFs also in one shot.

You didn't anwered to the main question about this topic: have you deploymentclient.conf in $SPLUNK_HOME\etc\system\local or in a dedicated app?

this is really important for the way (and the time) to change the new Deployment Server addressing.


The all 62 Universal forwarders are in 9.0.1 version and out of 62 UF's 14 are windows servers and 48 are linux servers.

My concern is with updating or modifying the deployment server details throughout all 62 universal forwarders, not with upgrading the Universal Forwarder itself. Instead of copying and pasting the add-on into each universal forwarder, is it feasible to make a new add-on and push it into all 62 of them at once?

If yes, Could you please guide me the process of creating addon and how to apply that add-on on linux OS and windows OS.

The deploymentclient.conf for the some servers is stored in \etc\system\local, and some servers are stored in dedicated apps.

Thanks & Regards,
Venkata Anuradha Neti.

[gcusello]

Hi @VenkataAnuradha,

at first I hint to move the outputs.conf from $SPLUNK_HOME/etc/system/local to a dedicated app in all your servers (both Linux and windows).

In this way you can change DS addressing in an easier way.

I didn't spoke of UF upgrade, but only about DS addressing.

Anyway, in my opinion the best approach is to create an add-on (called e.g. TA_Forwarders) containing three files:

• apps.conf: describing the Add-On
• deploymentclient.conf: addressing the Deployment Server,
• outputs.conf: addressing the Indexers.

If you already have a dedicated app, use it in all your servers.

the app can be the same on Linux and windows.

Ciao.

Giuseppe