Splunk cluster master migration

[rene_splunk]

I need to migrate our cluster master to a new machine. It currently has these roles:

• Cluster Master
• Deployment Server
• Indexer
• License Master
• Search Head
• SHC Deployer

I already migrated the License master role to the new server and it's working fine. I've been trying to follow the documentation here:
https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/Handlemanagernodefailure

From what I gather, I need to copy all the files in /opt/splunk/etc/deployment-apps, /opt/splunk/etc/shcluster and /opt/splunk/etc/master-apps, plus anything that's in /opt/splunk/etc/system/local.

Then add the passwords in plain text to the server.conf in the  local folder, restart Splunk on the new host and point all peers and search heads to the new master in their respective local server.conf files. 

Is there anything else that needs done or would this take care of switching the cluster master entirely? And is there a specific order in which to do things?

[rene_splunk]

The thing is, we've had this existing setup for years and never had any issues. Therefore, I had hoped to do a like for like swap to the new host, to prevent any further complications of building out more machines and making sure all hosts can connect, etc. However, I appreciate the replies and will try and implement some of these recommendations at least.

[PickleRick]

As Ismo said, it might have worked fairly OK but the rules for splitting roles between different hosts actually make sense. They let you easier scale each component according to your needs, you can - if needed - plan for HA/DR, you can plan migrations easier and you can easier debug stuff since separate roles don't interfere with each other.

[isoutamo]

Good to hear that it has worked, but you must remember that if you have any cluster, shc, DS etc. issue and needs support from splunk you will be in a deep s.... 😞 And Splunk's 1st requirements will be that you must fix your deployment architecture first and after that they can start to support it.

[PickleRick]

Wait, wait, wait a second...

Your single server has all these roles? Something is very wrong here.

OK. I assume that maybe it's only having those roles designated in Monitoring Console but still...

If a server seems to be both Cluster Master and Indexer, something is wrong with your setup. While you can sometimes in very small deployment mix some functionalities, a server which is supposed to be a CM, deployer, deployment server... that's a big no no!

Are you absolutely sure this server does all that?

[rene_splunk]

Correct, this is what is listed in the Monitoring Console as having all these roles. Our setup is as follows:

• 2 sites
• 9 Search Heads, clustered (5 in 1 site, 4 in the other)
• 8 indexers, clustered (split evenly)
• 2 heavy forwarders (site 1 only)
• 1 cluster master

I believe documentation mentions somewhere that in order to have a cluster master orchestrate clusters, it needs to take on the role of the cluster it's trying to orchestrate, ie search head or indexer. It doesn't actually fulfil those roles.

[PickleRick]

No.

In order to manage a cluster a server must be cluster master. And that's all. In rare cases (in small environments) it can fulfill other roles as well but in general case it shouldn't do anything else.

The SH deployer while it's theoretically also recommended to host it on a separate machine, it can be joined with another role (for example a small deployment server) since it's not very busy during normal SHC operations - it just pushes a SH config budle once in a while and that's it. Apart from that, the SHC is self-governing.

Generally speaking - indexers are those components that are indexing data. And that's it. No other components should have the role of indexer assigned to them (some people try to assign indexer role to HFs since they are processing the data and there is no separate role for them) and in a well-designed environment no other servers than indexers should do local indexing.

Search-heads are those components you can run searches against. You don't have to have searching capability on any other components than SHs and Monitoring Console (although you shouldn't use MC for "production" searching of course). In fact it's not uncommon to have CM/SH deployer/DS configured without webui (but yes, theoretically speaking, you can still dispatch searches via REST calls). I don't remember if all servers don't identify by default as search-heads since you can run a search against them - haven't set up a new MC for a while.

So CM should defintely not have at least some of those roles.

But just because your MC is apparently misconfigured it doesn't mean that your CM does all that.

Firstly find out what your CM really does.

[isoutamo]

@PickleRick already describes what those are and how those can combine. Here is Splunk's official matrix which server roles can be on same splunk instance https://docs.splunk.com/Documentation/Splunk/latest/Deploy/Manageyourdeployment

With your size of environment I strongly propose separate MC (monitoring console) to see and monitor your environment. That could be virtual or physical machine based on your environment. But it should be separate SH not any regular as there shouldn't be any other users than Admins also setting node as MC changes search behaviour as you set all other nodes as a search peers for it.

Personally I prefer to add all HFs as indexers into MC as that way I could check and monitor those via MC. BUT  strongly recommended that you must add separate custom group for those and also for your normal indexers which you start to use instead of indexer group. In that way you will see what you are expecting and those HFs values didn't disturb your statistics.

Where to put LM? That will depends on your environments and your update plans. Have you also a test environment which are using the same LM or not? Where you are sending its logs? etc.

One option is use CM or MC. But you must remember that LM is the 1st node which you should update to the new splunk version. See https://lantern.splunk.com/Splunk_Platform/Product_Tips/Upgrades_and_Migration/Upgrading_the_Splunk_... for updating splunk.

Anyhow when you are defining those roles for servers and add those to your environment I strongly recommended you to use CNAMEs or other way to separate role from physical node. In that way it's much easier later on to switch/migrate/what ever you need to a new physical node/site etc.

[rene_splunk]

I'm planning to move the MC role to the new server, however it needs to be configured as a search head to a multisite indexer cluster first. As far as I can gather from the documentation, https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/MultisiteCLI#Configure_the_search_heads

I just need to run the CLI command mentioned there on the new host, changing the master URI to my current master. However, after doing this and restarting Splunk on the new host, do I actually to go into my old host and set the MC mode to standalone before restarting or will that automatically be done when I continue setting up Distributed mode on the new host?

And finally, when do the clustered indexers get picked up as search peers in the MC, given I don't add them manually?

[isoutamo]

It's much easier and don't waist your resources if/when you set old "MC" as standalone MC after you have a new one up and running. Also you should remove unnecessary search peers from it's distributed search configuration.

On a new MC you must also add all needed nodes as a distributed search peer to see those on MC configuration tab and assign correct roles to those.

For clustered peers it's enough to add cluster master information and then it tells which nodes you have in your cluster at any particular time.

[rene_splunk]

That's interesting, because I added all the Search heads to the new MC, plus the current cluster master and I don't see the indexers listed on the distributed mode. I guess it may come after I've completed the setup of distributed mode, but I need to make the new instance a search head first according to the documentation, so I'll start there.

[isoutamo]

It's so long time when I have done this last time, that I cannot be 100% sure that it was that way. But at least https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ConfiguresearchheadwithCLI didn't mention that there is anything else what you need to do for individual peers.
Also this https://docs.splunk.com/Documentation/Splunk/latest/DMC/Addinstancesassearchpeers said "Do not add clustered indexers. If you are monitoring an indexer cluster and you are hosting the monitoring console on an instance other than the cluster manager, you must add the cluster manager as a search peer and configure the monitoring console instance as a search-head in that cluster."

[rene_splunk]

Yes, that was the documentation I was going on as well. As soon as I switched the old host to Standalone mode and configured Distributed mode on the new host, the indexers appeared, so that's the first part of the migration done anyway.

[isoutamo]

Hi

my old post about migrate the whole environment.

Based on splunk documentation, you cannot have all those roles on one server!

r. Ismo