Correct, see the note in this section: http://docs.splunk.com/Documentation/Splunk/6.3.1/Indexer/Configurethepeerindexes#View_the_indexes ... View more

You create indexes in indexes.conf, not server.conf. You update the file on the master node and then use the CLI command 'splunk apply cluster-bundle' to deploy the updates to the peer nodes. For details on updating configurations via that command, see: http://docs.splunk.com/Documentation/Splunk/6.3.1/Indexer/Updatepeerconfigurations (referenced from the 'configure peer indexes' topic as well) ... View more

The configuration bundle method is the only way to ensure that certain key configurations stay in sync across all peer nodes, which is necessary for the proper operation of the cluster. Regarding use of deployment server or third-party tools: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Keydifferences#Do_not_use_deployment_server_or_third-party_deployment_tools_with_cluster_peers ... View more

Regarding the number of indexers, or peer nodes, the only requirement is that you have at least as many as the replication factor. For details, see: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Systemrequirements#Required_Splunk_Enterprise_instances ... View more

This section of the documentation provides some details on cluster master/deployer compatibility: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Systemrequirements#Additional_roles_for_the_master_node ... View more

Answers to these questions are, by and large, readily available in the documentation. For information on indexer clustering system requirements, see: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Systemrequirements For search head clustering system requirements, see: http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/SHCsystemrequirements For basic information on how replication works in an indexer cluster, see: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Basicclusterarchitecture ... View more

Yes, the same method works to remove an app from the peers on an indexer cluster. ... View more

Also, if you do not explicitly set pass4SymmKey in the cluster master's [indexer_discovery] stanza, the master will use the value in its [general] stanza - either a value that you have explicitly set there or the default value. In either case, the forwarder's value in [indexer_discovery] must match that value. Therefore, the simplest way to deal with this is to set pass4SymmKey in the [indexer_discovery] stanza on the master, as well as on all the forwarders. ... View more

You must set the pass4SymmKey on the forwarders when using indexer discovery. The docs were incorrect on this issue, but have now been updated. See http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/indexerdiscovery#3._Configure_the_forwarders ... View more

That's not true. The cluster should be replicating changes made via the CLI. See: http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/HowconfrepoworksinSHC#Configuration_methods_that_trigger_replication ... View more

Thank you for the suggestion. We will make sure this gets added as a product enhancement request. ... View more

It's unclear from your question what the exact circumstances are for which you are trying to avoid a rolling restart. In search head clustering, rolling restart occurs only in two situations: After the deployer pushes updates to cluster members. (And, even then, not all updates require a restart. See http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/PropagateSHCconfigurationchanges#Push_the_configuration_bundle ) When you initiate a rolling restart with the splunk rolling-restart command. You say that you are using an "internal built application management tool". If that tool replaces the deployer functionality, then you can just restart members as you want, using whatever method you prefer. If that tool does not replace the deployer, then presumably your question is about how to avoid a potential rolling restart after pushing updates via the deployer. Unfortunately, there is no way to prevent a potential rolling restart, post-deployer push. Your plan to set percent_peers_to_restart to 0 won't do that, because, as the spec file for server.conf states, "regardless of setting, a minimum of 1 peer will be restarted per round." ... View more

sat94541 - The information you need to deploy a search head cluster and migrate settings to it from existing environments is already documented in full, in the Distributed Search manual. It's actually remarkably simple and straightforward. Be aware that the documentation contains information on requirements and such that is not included in the previous answer. To deploy a search head cluster and migrate existing settings to it, just follow these steps: Read the list of system requirements and other considerations, here: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCsystemrequirements Follow the procedure for deploying the cluster, here: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCdeploymentoverview Depending on whether you are migrating settings from a search head pool or from a standalone search head, see one of these two topics: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromsearchheadpooling http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Migratefromstandalonesearchheads Thank you. ... View more

Yes, you must set the search peers on each search head in the cluster. For details, read: http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/Connectclustersearchheadstosearchpeers ... View more

I think you might have to contact Support to get to the bottom of this issue. I suspect something went awry when you initially tried to deploy the file from the /users directory, and merely removing it from there somehow wasn't sufficient. ... View more

Did you delete the configurations from their locations under $SPLUNK_HOME/etc/shcluster/users? I think you will need to do that first, or at least delete the one that's directly under ../users. I would suggest creating a new subdirectory under ../apps (say "StandaloneConfigs") and putting outputs.conf , and any other standalone conf files, there, rather than adding them to an existing app. We'll look into clarifying the topic you reference, although on brief perusal I don't see any reference to the user directory in that topic. ... View more