How to install upgrade from version 5.0.1 & complete migration with complicated Splunk configuration?

[manderson7]

I've searched and seen a few migration threads, but my setup is a bit strange.

Two physical servers, 1st running splunk search head in /opt/splunk_sh, a splunk indexer in /opt/splunk_ind, and the deploy server under /opt/splunk_ds. The second physical server is running a splunk indexer in /opt/splunk_ind. All are running version 5.0.1.
I'd like to move onto 6.3.0, and so I've setup a VM for that purpose, and the plan is to move the search head function to the new vm. I'm now concerned about my order of operations:
1. stop splunk_sh on old server 1
2. copy /opt/splunk_sh to new server as /opt/splunk
3. install splunk6.3 on new server
4. startup new server

Once that's satisfied, upgrade the splunk indexers to 6.3?
Also, I need to move the deployment server off the old server to new vm, as well as license server. Uncertain about that order of operations as well.

Any advice is welcome here.

[woodcock · Accepted solution]

First of all, there is no reason to run a separate Splunk instance for your Search Head, Indexer, and Deployment Server on server 1; you should merge your configuration files and run a single instance on server 1 which will greatly uncomplicate things. In any case, much of this is arbitrary (can be shuffled) but this is how I would do it and it will definitely work:

SERVER 1:
splunk_ds/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_sh/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

SERVER 2:
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

NEW VM:
copy /opt/splunk_sh/* to new server as /opt/splunk/
MANUALLY (*know* what config files are important) merge /opt/splunk_ds/ intto /opt/splunk/.
upgrade to new version (copy new files over old files in same directory)
Install license keys here
Upgrade your deployment_client App (deploymentclient.conf) to point to this DS
Upgrade your deployment client App ([license] stanza with the master_uri setting in server.conf) to point to this LS
/opt/splunk/bin/splunk start # This is now Search Head and License Server and Deployment Server

SERVER 1:
remove /opt/splunk_sh/ and /opt/splunk_ds/ # already migrated
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

SERVER 2:
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

[manderson7]

Created the deployment app, deployed it to a client, but it's not taking precedence over the file in /opt/splunkforwarder/etc/system/local/deploymentclient.conf.

[manderson7]

NM on that, I'm having firewall issues...

[manderson7]

I dig it. Yeah, running each splunk type as a different instance on the servers really threw me the first time I saw it. not a fan.

My current plan, please comment if you feel like it 🙂
1. Setup new license/deployment vm
2. Test dev or ua box w/ new deploy server (create app w/ new deploymentclient.conf file to disable current file, pointing to deploy/license server), an aside: do the universal forwarders need the license information, or am I including that possibly in this deploy app for the search peers?
3. on legser1: splunk_sh/bin/splunk stop # tar up the directory and store it somehwere, just in case
4. also on legser1: splunk_ds/bin/splunk stop # tar up the directory and store it somehwere, just in case
5. copy data (easy or manual) from legser1:/opt/splunk_sh/ to newser1:/opt/splunk
6. install splunk 6.3 on newser1
7. spin up newser1, login and poke around to see what's broken?
8. once known healthy, continue with indexer upgrades?

[woodcock]

The only things that need to be pointed to the license server are the indexers but it is harmless to send the configuration stanza it to all servers (and it may make things simpler to package it that way).

[manderson7]

Thank you for the advice, and for helping calm my nerves on this seemingly massive project.

[woodcock]

So you are upgraded now?

[amiracle]

That sounds about right, and you can either setup a new license master and just add the indexer(s) to it as well as the search head(s). I don't think that it matters how you copy over the deployment server (before or after the license master). Additionally, you can just setup the new deployment server on the new box and just copy over the configs once the new box is up and running in the VM.

[manderson7]

I'm trying to avoid as much downtime as possible. I've found that the license master is running on the primary search head instance. I figure once I bring down the primary search head, the splunk installation is useless until I bring up
1. a license server
2. a search head

Is that accurate?
If that's the case, I'll need to have a license server ready, and have my license keys ready to import, as well as have my search head vm ready for duty.