This topic in the documentation discusses how summaries work in indexer clustering, with information on how to replicate summaries: http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Clustersandsummaryreplication ... View more

To replace the deployer, it should not be necessary to grab the runtime config from a cluster member, as the previous respondent was recommending. That is, unless you are also replacing the cluster members, but that does not seem to be your need. Rather, you merely need to configure the new deployer (to use the cluster's secret key and to set the SHC label), move the configuration bundle from the old deployer to the new deployer, and then point the cluster members to the new deployer. The configuration steps are outlined here: http://docs.splunk.com/Documentation/Splunk/6.6.3/DistSearch/PropagateSHCconfigurationchanges#Configure_the_deployer ... View more

The indexer cluster requires a dedicated search head. See http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Systemrequirements#Required_Splunk_Enterprise_instances The master node does have search head capabilities, but they can only be used for debugging purposes. See http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Systemrequirements#Additional_roles_for_the_master_node ... View more

Presumably that's because you're accessing the cluster members from a load balancer. To determine the instance, the best approach would be to log into the members directly. On the other hand, you can do as you did, which was to restart all instances. But it seems unnecessarily inconvenient to restart them all, when restarting just one would handle the need. ... View more

The system-level settings will only appear on those instances where you clicked "Show All Settings." ... View more

Actually, it is not necessary to perform a rolling restart of the entire cluster. You merely need to restart the instance(s) on which you previously exposed the system settings (via the "Show All Settings button). This is documented here: http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/HowconfigurationworksinSHC#The_Settings_menu ... View more

We only support rolling multisite upgrades from one minor version to the next. That is, we support 6.3 to 6.4, and 6.4 to 6.5, but not 6.3 to 6.5. With single version upgrades, the instructions in the documentation should work properly, and they have the advantage of allowing the cluster to return to a complete state more quickly than it would if you were to keep maintenance mode turned on for the entire duration of the upgrade process. There's no guarantee that leaving the cluster in maintenance mode for the entire upgrade process would always remediate issues arising from skip-version upgrades, in any case. The docs did not previously note the restriction regarding single-version upgrades. They do now: See http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Site-by-site_upgrade_for_multisite_indexer_clusters -- "If you have a multisite cluster, you can upgrade one site at a time, as long as you are upgrading across no more than one minor version (for example, from 6.4 to 6.5)." If you need to perform a skip-version upgrade, you have two choices: You can perform a non-rolling upgrade of the cluster, as described in http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Upgrade_a_6.x_indexer_cluster_to_a_later_version_of_6.x. You can perform a rolling upgrade in two steps. That is, to go from 6.3 to 6.5, you can upgrade all nodes from 6.3 to 6.4, followed by a second upgrade from 6.4 to 6.5. ... View more

Either one should work. However, note that maintenance mode does not currently persist across restarts of the cluster master. Also, maintenance mode stops fixup activity related to all peers, while the offline command stops fixup only for the peer being offlined. In terms of maintaining a valid cluster during long periods of maintenance, the ability to do so depends on the stability of your cluster as well as the search factor. For instance, if you have a search factor of 2 and another peer goes down during the maintenance period, you might temporarily lose searchability for some buckets. I'm not sure what you mean regarding your statement about forwarders. If you're using load-balancing as recommended, then the forwarder will just skip over the downed peer. ... View more

That restriction was removed with release 6.3. For current guidelines, see http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/SHCsystemrequirements#Search_head_clusters_running_across_multiple_sites ... View more

Short answer: No. They must reside on their own virtual or physical machines, as must all the other nodes in the clusters. In an indexer cluster, each node must reside on its own physical or virtual machine. See http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Systemrequirements#Machine_requirements This is also true for search head cluster members. See http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/SHCsystemrequirements#Machine_requirements_for_cluster_members ... View more

As the cited doc states, "The only way to delete an app-level baseline knowledge object is to redeploy an updated version of the app that does not include the knowledge object." So, that is the workaround. ... View more

The docs have been updated to read, "When you upgrade a 6.x indexer cluster, such as 6.2, to a later 6.x cluster, such as 6.3 or 6.4, you must take down all cluster nodes. " ... View more

It is correct that you do not need to set up the deployer connectivity until the first time that you need to use the deployer. However, the deployer is the only supported method for deploying apps and other configurations to cluster members. So, sooner or later, you will need to set up the deployer. You cannot deploy an app until you do so. For this reason, we recommend that you configure the deployer and connectivity to the deployer at the same time that you initially configure the cluster itself. There is usually no reason to wait until later. ... View more

I downvoted this post because the problem that the user had was not with establishing connectivity between search heads and search peers. rather, the problem was with the query used to test connectivity. the accepted answer provided a procedure tht, if followed by others, would cause them to completely rebuild their search head cluster, among other issues. this answer should be removed, as it could easily mislead anyone who does have search peer connectivity issues. ... View more

You might want to double check the management port on your indexer, to make sure that it's 8089. See http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Changedefaultvalues#Use_Splunk_Web_2 ... View more