I have got a complicated task of consolidating two standalone search heads and a single search head cluster (4 nodes) all into a single search head cluster of 3 nodes.
Can someone please advise what would be the most efficient and correct method to accomplish this ?
What do you mean by "consolidating" since you have 6 hosts and want to reduce this number to 3?
Meaning migrating the search artifacts of those 3 search heads(1 being the SHC) into the new single SHC.
Basically, right now there are people using SHC for Reporting, 1 SH for purely dashboards, and 1 for adhoc searching, all of which are on-prem. I want them to just use one SHC for all those scenarios. This SHC will be newly built in self managed Splunk cloud environment.
You don't have "3 search heads(1 being the SHC)". You have 6 SHs and have to check what deployment mode you use on the SHC 🙂
It's an annoying process because you have to migrate all roles, users (you might have this already "done" if you use external auth), apps, user data (which can be tricky if you want to merge settings from different SHs into one SHC). There's no automatic way to do that, unfortunately. It's a painful and tedious manual process.
Yes, my bad. From the point of view of unique search artifacts, I meant to say there are 3 SHs (considering nodes in SHC will contain same artifacts.)
@PickleRick wrote:user data (which can be tricky if you want to merge settings from different SHs into one SHC)
Thats the key part I am concerned about and wondering how I would go about properly migrating them.
Can you please share how the manual process of this would like ?
I am aware its mainly a matter of copying over the etc/apps and etc/users directories. Luckily, there are no conflicting apps but there are definitely same users across all these SHs. Mainly, I 'd appreciate if you can tell me whats the best way do that ?
The general idea is that you indeed copy all apps over but as I said - it can be tricky depending on your push mode.
We use merge_do_default so what we did was gather all the configs from the search heads and simply copied them over to the deployer. With some exceptions. You should not overwrite default on built-in apps (like search) so any changes our users did to the search app we simply migrated to a new app.
The users - if I remember correctly, you can just copy over.
Thank you so much for sharing your valuable experience.
In the case of users, how did you ensure config belonging to the same user residing on two different standalone SHs is migrated successfully without losing any data ?
Our case was slightly different (we were simply creating a new SHC based on an existing one). But we simply copied over all users' directories from one SH from old cluster to all SH-s in the hew cluster if I recall correctly