SHC: file integrity differences and replication fault

[Jamie]

Hello.  

I have two, possibly related, problems with my three node SHC (version 8.2.2).

One or both may stem from using the Deployer to push out changes to app.conf for the default apps (I was trying to disable checks for updates).

1. On the DMC, each SHC node reports file differences in app.conf for default apps.  Also some files are listed as missing for splunk_essentials_8_2.

I tried correcting this by reversing the work undertaken with the Deployer. Without sucesss. I then decided to make the same changes on each node manually.

2. The SHC nodes report:

[date] ERROR ConfReplicationThread [13247 ConfReplicationThread] - Error pulling configurations from captain=https://shc_1:8089, consecutiveErrors=74 msg="Application does not exist: 504df959a582d73": Search head cluster member (https://shc_2:8089) is having problems pulling configurations from the search head cluster captain (https://shc_1:8089). Changes from the other memers are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.

These messages are stopped with:

bin/splunk resync shcluster-replicated-config

However, the problem returns if the SHC nodes are restarted.

I would be grateful for your help in fixing these problems.

[isoutamo]

Hi
When you are saying "default apps" what you are actually meaning (I hope that not search, launcher etc.)?
r. Ismo

[Jamie]

Hello.

Yes, I added:

check_for_updates = 0

to app.conf for the following on the Deployer and pushed this out:

alert_logevent/local
alert_webhook/local
appsbrowser/local
introspection_generator_addon/local
journald_input/local
launcher/local
learned/local
legacy/local
python_upgrade_readiness_app/local
sample_app/local
search/local
splunk_archiver/local
splunk-dashboard-studio/local
splunk_essentials_8_2/local
SplunkForwarder/local
splunk_gdi/local
splunk_httpinput/local
splunk_instrumentation/local
splunk_internal_metrics/local
SplunkLightForwarder/local
splunk_metrics_workspace/local
splunk_monitoring_console/local
splunk_rapid_diag/local
splunk_secure_gateway/local
user-prefs/local

[isoutamo]

Splunk's instructions are: Never ever deploy those default apps with Deployer to SHC node!

I'm not sure if there is an easy way to remove this deployment dependencies or not for those default apps. Usually when you remove any apps from deployer's .../etc/shcluster/apps then it remove WHOLE app from all SHC nodes and this is definitely something which you don't want!

I propose that you should contact to splunk support, if they have any reasonable way to help you or is the only way to create your SHC from scratch with those other apps.

r. Ismo