Migrate from indexer cluster to standalone instance

[pcsegal1]

Hi,

I have a legacy Splunk Enterprise cluster that consists of:

• 1 cluster master
• 3 indexers, forming an indexer cluster
• 1 search head
• 1 license master

This cluster will stop receiving data. I need to downgrade it from cluster to standalone, and I need to preserve its existing data in such a way that it remains searchable.

That is, I need to downgrade this cluster to only one instance: a single standalone instance that contains the same data as the indexer cluster.

Is this possible? What steps should I perform?

[Bellthazor]

@woodcock I am new to splunk and am not sure how to create the script referenced for converting `single-site buckets` to `unclustered buckets` and wanted to know if you could provide a reference/example for this.  Thanks in advance for any assistance!!

[woodcock]

You will have to contact splunk support for that.

[woodcock]

The easiest thing to do is to upgrade to smarstore and then just turn off 2 of your indexers. This is CAKE.
Alternatively. If you are NOT multi-site:

0: Create a script that can convert `single-site buckets` to `unclustered buckets` (this is pretty easy).
1: Set `RF=1/SF=1`; wait for things to settle.
2: Disable `Indexer Discovery` feature, reverting to traditional list of Indexers, but specifying all 3 Indexers.
3: `Remove Excess Buckets` from the CM; there is now only 1 copy of each bucket.
4: SEARCH OUTAGE IS ABOUT TO BEGIN: shutdown Search Heads
5: Shutdown the indexer that is to become the lone survivor (incoming data going to other indexers).
5a: Enlarge disk volumes if need be (probably).
5b: Run script to convert `single-site buckets` to `unclustered buckets`.
5c: Remove `Indexer Clustering` settings from this indexer.
5d: Restart Splunk; incoming data will now go to `unclustered buckets`.
6: Update `outputs.conf` with traditional list of Indexers and specify ONLY the 1 Indexer that will be the lone survivor.
7: Shutdown the other indexers.
8: Run script to convert `single-site buckets` to `unclustered buckets`.
9: Copy the buckets to the lone survivor.
10: Restart the lone survivor Indexer.
11: Restart the Search Head(s).
12: Trash the other Indexers and the Cluster Master.

If you ARE multi-site, you will have to downgrade to single-site, then downgrade to a single Indexer which is much the same as expanding your Indexer cluster.

[richgalloway]

The path from single indexer to indexer cluster is well-known. The path from cluster to single is not. Few downgrade like that. You can go to singler-indexer cluster, however. By staying clustered it's easy to add nodes when you need to.

Here are the steps I would take.
1) Change your replication and search factors to 1
2) Change all servers that send data to Splunk to send only to indexer 1. This should be matter of pushing a new outputs.conf file.
3) Put indexers 2 and 3 into manual detention using this command on each:

splunk edit cluster-config -auth <username>:<password> -manual_detention on

This stops the indexers from accepting data and from replicating data from indexer 1.

4) Take indexer 2 off-line. The --enforce-counts option tells the cluster master to move all primary and searchable buckets to another indexer (which will be indexer 1 since it is the only one not in detention).

splunk offline --enforce-counts

5) Wait for buckets to move and for indexer 2 to stop.
6) Repeat steps 4 and 5 with indexer 3.
7) Remove indexers 2 and 3 from the list of search peers on the SH
8) Decommission indexers 2 and 3.

To reduce the number of instances further, make the SH your license master.

[rajashaey]

@richgalloway but in your steps i didnt see anything to convert bucket renames? Its not required?

[richgalloway]

There is no need to rename buckets in this case.

[pcsegal1]

Thank you. I just wanted to clear up a doubt: if the indexer can support direct searching, then, after the procedure you outlined, if I then remove the indexer from the cluster, wouldn't it effectively become a standalone instance?

[richgalloway]

If you remove the indexer you will remove all of your data.
To further condense your installation, you will need to rename all of the data buckets to the non-clustered name format to become a non-clustered indexer. See @woodcock's answer for details. You will also need to copy all of your knowledge objects from the SH to indexer so the indexer can become the standalone instance.

[pcsegal1]

@richgalloway - I have a follow-up question. Let's say that the cluster's Enterprise license expires, and all instances are downgraded to Free. Since clustering and distributed search are not supported in Free, does that mean that all data is automatically removed? Or is the data still preserved?

[richgalloway]

Data is not removed when licenses expire.

[pcsegal1]

@richgalloway - Thank you. Just to clarify, the reason for the question was that, when downgrading a cluster to Free, I assume all instances would automatically become standalone. Is that correct? So, I imagined this license downgrade would end up in the same complex scenario of going from cluster to standalone. Would any complex procedure be required in this case as well?

[richgalloway]

I can get you from a three-node cluster to a single-node cluster, but I'm not sure it's possible to go back to a standalone instance. Are you sure that's what you want?

[pcsegal1]

@richgalloway Generally speaking, what I need is to reduce the number of instances as much as possible (ideally to 1) in order to save costs while still keeping the legacy data accessible. If going back to a standalone is very tricky or impossible, I think downgrading to a single-node cluster could be a good compromise. Could you please share the procedure? Thank you!

[richgalloway]

For going from standalone to clustered, Splunk recommends engaging Splunk Professional Services. Going the other way is at least as complicated so PS should be considered there, too.