Getting Data In

distsearch.conf on Search Head cluster

nunoaragao
Explorer

Hi Splunkers,

Long time ago we setup a SH cluster, and added search peers using CLI

Some time later we changed the setup and began setting the search peers via an App pushed from the deployer. All $SPLUNK/etc/system/local distsearch.conf files were purged, and the app contains some peers on distsearch.conf and clustered indexers on server.conf

Recently we found one cluster member stubbornly kept re-creating a distsearch.conf on system/local that overrided the cluster's configs ( pushed via the App ). Removing the file and doing a rolling restart, the file showed up again. Cleaning the KV Store and adding that member to the cluster restored the rogue distsearch.conf again. We also deleted *bundle files on $SPLUNK/var/run.

Following the steps described as "Add a member that was both removed and disabled" here:
https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Addaclustermember

We found when calling "splunk add shcluster-member" on the Captain, the "new" member's system/local path was purged from some files and re-created .. with the rogue distsearch.conf we were trying to get rid.
And this event showed up on _internal
ERROR ApplicationUpdater - Error reloading : handler for distsearch (access_endpoints /search/distributed/bundle-replication-files, /search/distributed/peers)

Any ideas ? The final solution was to nuke the VM and re-create from scratch.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...