I have a legacy Splunk Enterprise cluster that consists of:
This cluster will stop receiving data. I need to downgrade it from cluster to standalone, and I need to preserve its existing data in such a way that it remains searchable.
That is, I need to downgrade this cluster to only one instance: a single standalone instance that contains the same data as the indexer cluster.
Is this possible? What steps should I perform?
I can get you from a three-node cluster to a single-node cluster, but I'm not sure it's possible to go back to a standalone instance. Are you sure that's what you want?
@richgalloway Generally speaking, what I need is to reduce the number of instances as much as possible (ideally to 1) in order to save costs while still keeping the legacy data accessible. If going back to a standalone is very tricky or impossible, I think downgrading to a single-node cluster could be a good compromise. Could you please share the procedure? Thank you!
For going from standalone to clustered, Splunk recommends engaging Splunk Professional Services. Going the other way is at least as complicated so PS should be considered there, too.
The path from single indexer to indexer cluster is well-known. The path from cluster to single is not. Few downgrade like that. You can go to singler-indexer cluster, however. By staying clustered it's easy to add nodes when you need to.
Here are the steps I would take.
1) Change your replication and search factors to 1
2) Change all servers that send data to Splunk to send only to indexer 1. This should be matter of pushing a new outputs.conf file.
3) Put indexers 2 and 3 into manual detention using this command on each:
splunk edit cluster-config -auth <username>:<password> -manual_detention on
This stops the indexers from accepting data and from replicating data from indexer 1.
4) Take indexer 2 off-line. The
--enforce-counts option tells the cluster master to move all primary and searchable buckets to another indexer (which will be indexer 1 since it is the only one not in detention).
splunk offline --enforce-counts
5) Wait for buckets to move and for indexer 2 to stop.
6) Repeat steps 4 and 5 with indexer 3.
7) Remove indexers 2 and 3 from the list of search peers on the SH
8) Decommission indexers 2 and 3.
To reduce the number of instances further, make the SH your license master.
Thank you. I just wanted to clear up a doubt: if the indexer can support direct searching, then, after the procedure you outlined, if I then remove the indexer from the cluster, wouldn't it effectively become a standalone instance?
If you remove the indexer you will remove all of your data.
To further condense your installation, you will need to rename all of the data buckets to the non-clustered name format to become a non-clustered indexer. See @woodcock's answer for details. You will also need to copy all of your knowledge objects from the SH to indexer so the indexer can become the standalone instance.
@richgalloway - I have a follow-up question. Let's say that the cluster's Enterprise license expires, and all instances are downgraded to Free. Since clustering and distributed search are not supported in Free, does that mean that all data is automatically removed? Or is the data still preserved?
@richgalloway - Thank you. Just to clarify, the reason for the question was that, when downgrading a cluster to Free, I assume all instances would automatically become standalone. Is that correct? So, I imagined this license downgrade would end up in the same complex scenario of going from cluster to standalone. Would any complex procedure be required in this case as well?