If I add or remove a peer node into/from a existing search head cluster or indexer cluster, do I need to restart splunkd? If I do it via CLI, is it required to restart splunkd? how about doing it via editing configuration file?
I went through http://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Restartthecluster but it seems to be vague.
Indexing and Search Head Cluster peer removal process is different... Both of them, the options from the GUI are limited..
For removing a peer from a Indexing Cluster, you need to run a splunk offline from the peer you want to remove. Once you do that and the peer gracefully removes itself from the Cluster, you can either restart the CM for it to disappear (not recommended) or follow the process listed below..
See the following Documentation :
https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Takeapeeroffline
https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Removepeerfrommasterlist
For a SHC, peer removal is a bit less convoluted.. essentially from the peer you want to remove, you need to run the splunk remove shcluster-member, or from another peer you can run the following : splunk remove shcluster-member -mgmt_uri https://hostnameofpeertoremove:8089
See docs :
https://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Removeaclustermember
Note that this can have adverse effects on your SHC, especially when you dont have enough peers remaining to elect the new captain.
If you follow the above processes, which are the recommended methods of doing this, you do not need to restart Splunk. You could edit the configuration files, but doing so would require a restart of Splunk.
If you do it via CLI , you don't need to restart Splunkd. However, if you do it via updating configuration files, you need to restart the Splunkd services.