Error pulling configurations from the search head cluster captain - Caused by missing app in deployer?

[ben_leung]

Curious how this was caused? Is it because the app was removed in the deployer?

Banner message:

Error pulling configurations from the search head cluster captain (https://myhost:8089); consider performing a destructive configuration resync on this search head cluster member

In splunkd.log:

04-01-2015 22:24:59.846 +0000 WARN  ConfReplicationThread - Error pulling configurations from captain=https://myhost:8089, consecutiveErrors=16057: Error in fetchLookupTableContents, asset_id=229e28b611b458fe1c05b7ed1d8e83cb0f8b9481: Non-200 status_code=500: Application does not exist: sos

[ben_leung · Accepted solution]

Okay seems like that was the issue, placed the sos app back in the deployer to distribute. Logs are not showing any errors.

[theunf]

Need to confirm ... how about those apps... they are system related (6.2.3) :

learned/
sanitycheck/
splunk_datapreview/

What about if IU find these apps on the shcluster/apps at the deployer ???

If I remove them, they get removed on the Search Heads, right ?
To fix this, on all search heads : a manual backup
Then remove the folders from shbundle, deploy from deployer;
And then return backup on each search head ?

[theunf]

I have other odd error , but in the other way, from the Search Head to the captain :

splunkd.log:05-25-2015 21:13:18.267 -0300 WARN ConfReplicationThread - Error pushing configurations to captain=https://:8089, consecutiveErrors=1: Error in acceptPush: Non-200 status_code=400: Error parsing JSON response: String value too long

and

metrics.log:05-25-2015 21:18:56.132 -0300 INFO Metrics - group=captainstability, stable_follower_pct=100, stable_captain_pct=0, num_polled_captain=0, num_polled_follower=155, num_polled_candidate=0, upgrades_to_captain=0, downgrades_from_captain=0, captain_changes=0
metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_counts, name=shccaptain_artifact, to_fix_rep_factor=0, to_fix_added=0, to_fix_removed=0, to_fix_total=0, count=0
metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_seconds, name=shccaptain_service, seconds=0.000