Search Head Deployer and Search Head Cluster Sync Missing so cant deploy Apps

[arunsoni]

Hello,

I have a question about sh deployer and search heads. We have three search heads within a cluster and for some reason at some point of time deployer connection got disconnected and now I am trying to connect it. Let me know what need to be done ? Is it just we need to match password of all search heads with deployer.

Configurations I currently see:

On Search head(1/2/3): /opt/splunk/etc/system/localserver.conf

[shclustering]
conf_deploy_fetch_url = https://XXXXXX:8089
disabled = 0
mgmt_uri = https://XXXXXXX:8089
replication_factor = 2
shcluster_label = shcluster1
id = 1F81D83B
manual_detention = off

Deployer : /opt/splunk/etc/system/localserver.conf

[shclustering]
shcluster_label = shcluster1
pass4SymmKey = XXXXXXX

Thanks in advance for your help!

[gcusello]

Hi @arunsoni ,

as you can read at https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/AboutSHC

in a SH Cluster Deployer is relevant only for deployng new apps and updates.

During normal running, Deployer doesn't partecipate to the activities because SHs reply configurations and lookups data by themselves managed by one of the SHs elected by themselves "Captain".

Ciao.

Giuseppe

[isoutamo]

One thing which I “found” (fortunately in test with backups). If your deployer is down enough long time your SHC members lost all apps which have deployed by deployer!

Have you lost your deployer or only connection between it and members?

If first then it should be enough that you restore the connectivity between those. Of course you must ensure that deployer still has all those apps which have previously deployed to members.

If you have lost whole node and you haven’t backup and you must build it from scratch then there are some things which you must check and update before you can put it back online.

• Ensure that you have those apps there what you have previously deployed with same configuration on place.
• Ensure that lookups are correct.
• check that deployment modes are globally and app level correctly setup (how local and default are transferred into members)
• check that how lookups should push into members override or keen member version?
• check if members have same splunk.secrets and if copy this into deployer before start it first time
• then those what you and @gcusello already mentioned 
• check that all nodes have same time!

Maybe something else? If you could do and test this on test environment, do it first and check what are issues which arise after deployer is back online.

If you haven’t the definite you must take backup of all those nodes when they are offline! And include kvstore backup too.

I would like to hear how this succeed after you have done it!

[arunsoni]

@isoutamo @gcusello .

We have a deployer but used few years back and from last couple of year each app we are deploying on search heads manually without using deployer. I recently tried to deploy app using deployer and see there was password mismatch.

1. Now If I set the password on a deployer and match with all search heads, Is it okay if I push an app or do I see any issues ?

2. If I push apps from deployer will there be any issues with apps exist on search heads not on deployer ?

3. After deployment Will there be only changes for apps existing on both deployer and search head ?

[isoutamo]

Ok, basically your environment is broken and it doesn’t fulfill any Splunk’s requirements for SHC! I said that sooner or later you must rebuild it from scratch. I suggest that you will do it as soon as possible. Now you have those apps, congratulations, data etc still there and you could utilize those with a new environment without issues. But if your environment will collapse then you probably lose at least some of that data.

if you don’t know how to do this, pleas ask some local Splunk partner or Splunk PS to do it. There are good instructions on splunk docs how you could migrate from individual SH to SHC. You could use those as starting point,  but probably you need to modify those somehow depending how your “SHC” is currently working.