Expand a single node to a cluster

[desmando]

I'm trying to take a single node Splunk Enterprise system and expand it to a cluster with an additional search head and indexes.

I copied the existing install to a new system and that worked perfectly.

Then I added the cluster manager and indexes and all of the settings that were in the old system that were copied to the search head were gone.

I'm assuming that I put the copy of the single node into the wrong role, but I'm not sure which role I should have picked.

[PickleRick]

Wait. What do you mean by "expand to a cluster"? And what are you trying to achieve?

I understand that initally you have an all-in-one installation. What architecture are you aiming at?

Cluster (unless explicitly referenced to as SH cluster) typically means cluster of indexers with a Cluster Manager. For that you need at least a single separate SH.

So for a clustered installation you need at least three nodes - one SH, one CM and at least one indexer.

The first thing to do if you indeed have an AIO setup would be to add an external SH and turn your existing server into a pure indexer.

After you have done that you might think of converting the indexer to a cluster node.

[desmando]

Sorry for not getting terms right. So I started with an AIO. I added a Cluster Manager and Two Indexes. I connected the AIO to this as the Search Head.

In that process I lost all of the settings and data that were in the AIO.

[PickleRick · Accepted solution]

I'm not sure what you mean by "settings" but since your AIO had all the indexed data and you've spun up new empty indexers that's logical that your SH will search the empty indexers.

The proper way to expand from a single AIO server is either as @isoutamo wrote (which is a bit more complicated to do as a single migration. or the other way:

1) Add another host as search head, migrate search-time settings there. Leave your old server as indexer. Verify if everything is working properly.

2) Add a CM, add your indexer as a peer to the CM. You might either set RF=SF=1 for starters and then raise it later when you add another peer or you can add another indexer at this step. The trick here is that your already indexed data is not clustered and while it should be searchable it will not get replicated.

[isoutamo]

When your starting point is AIO and you want to go single sh + indexer cluster and you want to keep your old data then the steps in high level are 

1. install cm and configure it 
2. add current AIO node as 1st peer
3. add 2nd peer
4. add a new SH
5. copy needed apps etc from AIO into new SH

Please check the exact steps and how to do those from @gcusello ‘s pointed document. There are detailed instructions how configure your CM, how to add peers, when and how to copy apps, when to remove unnecessary apps from old AIO node before use it as search peer etc.

[isoutamo]

Here is Splunk Validated Architectures https://docs.splunk.com/Documentation/SVA/current/Architectures/TopologyGuidance

[desmando]

I've seen that, but I don't see in it the right way to move between topologies.

[gcusello]

Hi @desmando ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello]

Hi @desmando ,

you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/9.4.0/Indexer/Migratenon-clusteredindexerstoaclusterede...

in few words:

• install the same Splunk version on the new Indexers and Cluster Manager,
• configure the CM as Cluster Manager node,
• configure IDXs as peer nodes,
• modify the IDX configurations for a cluster,
• deploy the configurations of the old IDX to both the peers using the CM,
• configure the SH to access the cluster.

In the CM, you should see both the IDXs and all the indexes replicated.

Remember that only new data are replicated between the IDXs, old ones aren't replicated,

To replicate also old data, you need a Splunk Professional Services or a Certified Core Consultant.

Ciao.

Giuseppe