Curious how this was caused? Is it because the app was removed in the deployer?
Error pulling configurations from the search head cluster captain (https://myhost:8089); consider performing a destructive configuration resync on this search head cluster member
04-01-2015 22:24:59.846 +0000 WARN ConfReplicationThread - Error pulling configurations from captain=https://myhost:8089, consecutiveErrors=16057: Error in fetchLookupTableContents, asset_id=229e28b611b458fe1c05b7ed1d8e83cb0f8b9481: Non-200 status_code=500: Application does not exist: sos
Need to confirm ... how about those apps... they are system related (6.2.3) :
What about if IU find these apps on the shcluster/apps at the deployer ???
If I remove them, they get removed on the Search Heads, right ?
To fix this, on all search heads : a manual backup
Then remove the folders from shbundle, deploy from deployer;
And then return backup on each search head ?
I have other odd error , but in the other way, from the Search Head to the captain :
splunkd.log:05-25-2015 21:13:18.267 -0300 WARN ConfReplicationThread - Error pushing configurations to captain=https://:8089, consecutiveErrors=1: Error in acceptPush: Non-200 status_code=400: Error parsing JSON response: String value too long
metrics.log:05-25-2015 21:18:56.132 -0300 INFO Metrics - group=captainstability, stable_follower_pct=100, stable_captain_pct=0, num_polled_captain=0, num_polled_follower=155, num_polled_candidate=0, upgrades_to_captain=0, downgrades_from_captain=0, captain_changes=0
metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_counts, name=shccaptain_artifact, to_fix_rep_factor=0, to_fix_added=0, to_fix_removed=0, to_fix_total=0, count=0
metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_seconds, name=shccaptain_service, seconds=0.000