Hi Splunkers,
Long time ago we setup a SH cluster, and added search peers using CLI
Some time later we changed the setup and began setting the search peers via an App pushed from the deployer. All $SPLUNK/etc/system/local distsearch.conf files were purged, and the app contains some peers on distsearch.conf and clustered indexers on server.conf
Recently we found one cluster member stubbornly kept re-creating a distsearch.conf on system/local that overrided the cluster's configs ( pushed via the App ). Removing the file and doing a rolling restart, the file showed up again. Cleaning the KV Store and adding that member to the cluster restored the rogue distsearch.conf again. We also deleted *bundle files on $SPLUNK/var/run.
Following the steps described as "Add a member that was both removed and disabled" here:
https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Addaclustermember
We found when calling "splunk add shcluster-member" on the Captain, the "new" member's system/local path was purged from some files and re-created .. with the rogue distsearch.conf we were trying to get rid.
And this event showed up on _internal
ERROR ApplicationUpdater - Error reloading : handler for distsearch (access_endpoints /search/distributed/bundle-replication-files, /search/distributed/peers)
Any ideas ? The final solution was to nuke the VM and re-create from scratch.
