First, there are no plans to add views to the S.o.S app that will provide visibility & introspection for Search Head Clustering. The good news, however, is that the Distributed Management Console ships with Search Head Clustering dashboards as of Splunk Enterprise 6.3! Most notably, the very first view titled "Status and Configuration" provides an overview of your Search Head Cluster(s) and will show a list of cluster members and point out the captain. Do note that this information is not available directly from search-heads that are members of the cluster as the DMC is not supported there - you'll need to set up an instance outside of the cluster (typically, the app deployer) to be the DMC and monitor the cluster from outside. ... View more

Oh good, let me convert my comment to an answer then. ... View more

I would suggest to set up (and adjust as needed) the "Total license usage near daily quota" platform alert that ships with the Distributed Management Console. ... View more

I see! One issue here is that the data from which the forwarder asset table is constructed is already indexed in the internal index. The DMC will _not clean that up - you don't want that anyway. I think that your main (and maybe only) option here is to identify those forwarders that have different GUIDs but identical values for server.conf / serverName and to correct that. In order to do that, you should manually search your tcpin_connection events like so: index=_internal source=*/metrics.log* group=tcpin_connections (cooked OR cookedSSL) | stats dc(hostname) AS hostname_count values(hostname) AS hostnames values(host) values(sourceHost) AS source_hostnames by guid | where hostname_count > 1 ... View more

Note that the DMC also comes with a built in alert to proactively detect missing forwarders - see http://docs.splunk.com/Documentation/Splunk/6.3.0/DMC/Platformalerts ... View more

1: That particular lookup file is maintained by a scheduled search named "sos_refresh_splunk_forwarders_cache" and it is updated every 15 minutes by default. 1bis: I would expect the warnings to go away as soon as you have a lookup file with just the header, or one with forwarder records as well. 2: Good eye! S.o.S is the ancestor of the Distributed Management Console and some people who developed the former (such as yours truly) have participated in the engineering of the latter. In that sense, you will find them occasionally related. And most definitely, the DMC is poised to replace S.o.S as it delivers most of the same content and much, much more! ... View more

That particular lookup serves as an asset table for forwarder instances that are known to the S.o.S app. The columns it contains are described in $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv.spec , which shares the same columns. The main reason why we cannot ship that file by default with the S.o.S app is simply that it would overwrite what your instance has already generated, as there is no concept of "default" and "local" spaces for lookup files. By all means, do add a header line to the lookup file to make the message go away. ... View more

You can edit the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv file manually and set the value of the "sos_server" field to the value of the "host" field reflected by events recorded in the "sos" and "_internal" indexes for that particular instance. ... View more

I don't think that we have identified a limit in the number of search-heads that can be attached to an indexer cluster. Of course, if those search-heads are going to be members of a search-head cluster, you'd want to keep that number at a maximum of 50 if you want to remain within the supported boundaries. ... View more

By the way, just to disambiguate: Are you talking about an indexer cluster, a search-head cluster or a combination of both? ... View more

I'm sorry to hear this suggested work-around did not function. I would like to strongly encourage you to open a support case so that we can look into this issue in more detail and identify if there is a new defect to be fixed here. ... View more

I would also suggest to use the Search-Head Clustering views of the Distributed Management Console to this effect. Notably, the "General Status & Configuration" view shows a history of recent captain elections. ... View more

This issue has been identified as a product defect - internal reference: SPL-92633. The problem is quite simply that some DMC actions (typically, configuration changes) are hard-coded to run lookup-manipulating searches as the "admin" user, which of course fails if the user in question has been renamed. The work-around (and actually, the fix too) is to leverage the dispatchAs = user property in savedsearches.conf (new to 6.2) which allows a saved search to be run as the invoking user instead of the owning user when called. Work-around steps: Add the dispatchAs = user key to the DMC Asset - Build Full saved search stanza in $SPLUNK_HOME/etc/apps/splunk_management_console/local/savedsearches.conf Restart Splunk or hit the /debug/refresh UI endpoint Run DMC setup again ... View more

Actually, I was wrong: The fix for this issue did not make it into 6.3 which explains why you are still seeing it! I will explain how to work around this problem in an answer. ... View more

There was a specific issue with the DMC setup and renamed admin accounts that was fixed in 6.3. Can you describe in detail what interactions with the DMC are no longer working and how that manifests itself? ... View more