Getting Data In

distsearch.conf on Search Head cluster

nunoaragao
Explorer

Hi Splunkers,

Long time ago we setup a SH cluster, and added search peers using CLI

Some time later we changed the setup and began setting the search peers via an App pushed from the deployer. All $SPLUNK/etc/system/local distsearch.conf files were purged, and the app contains some peers on distsearch.conf and clustered indexers on server.conf

Recently we found one cluster member stubbornly kept re-creating a distsearch.conf on system/local that overrided the cluster's configs ( pushed via the App ). Removing the file and doing a rolling restart, the file showed up again. Cleaning the KV Store and adding that member to the cluster restored the rogue distsearch.conf again. We also deleted *bundle files on $SPLUNK/var/run.

Following the steps described as "Add a member that was both removed and disabled" here:
https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Addaclustermember

We found when calling "splunk add shcluster-member" on the Captain, the "new" member's system/local path was purged from some files and re-created .. with the rogue distsearch.conf we were trying to get rid.
And this event showed up on _internal
ERROR ApplicationUpdater - Error reloading : handler for distsearch (access_endpoints /search/distributed/bundle-replication-files, /search/distributed/peers)

Any ideas ? The final solution was to nuke the VM and re-create from scratch.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...