We are building a single-site pilot environment with the following layout:
1 x Deployment and License Manager
3 x Search heads (configured in a SH cluster)
3 x Indexers
1 x Indexer Cluster Master
We have the indexers set up and sharing data and we even have a test forwarder sending data to an index on those clusters, however, when we connect to the search heads (either directly or via our load-balanced IP in front of them) the search does not see any data whether from the internal indexes or the external test index we built for our test data.
We ran the
/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list command and listed all of our search heads. We also ran the
/opt/splunk/bin/splunk edit cluster-config-mode searchhead -master_uri on the captain.
We've read through the http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/SHCandindexerclusterdocumentation but we're not clear whether we need to run
edit cluster-config -mode searchhead -master_uri command on each of the search heads. That is what we've done, but it doesn't appear to resolve the issue.
Here is the output of the /opt/splunk/bin/splunk list cluster-config command:
config access_logging_for_heartbeats:1 cxn_timeout:60 disabled:0 forwarderdata_rcv_port:? forwarderdata_use_ssl:0 heartbeat_period:0 heartbeat_timeout:60 master_uri:https://[IP of the index cluster master]:8089 max_peer_build_load:5 max_peer_rep_load:5 mode:searchhead multisite:false percent_peers_to_restart:10 ping_flag:1 quiet_period:60 rcv_timeout:60 rep_cxn_timeout:60 rep_max_rcv_timeout:600 rep_max_send_timeout:600 rep_rcv_timeout:60 rep_send_timeout:60 replication_factor:3 replication_use_ssl:0 restart_timeout:60 search_factor:1 search_files_retry_timeout:600 send_timeout:60 site:default
Also enabled the DHC and confirmed that we are not seeing a search head cluster defined there either.
Hi joshuabiggley, You will want to setup each index cluster slave as a search peer on each search head cluster member. I believe this will resolve your issue. Please let me know how it works for you! 😄
We had already added the cluster slaves as search peers on each search head cluster member. We used the command below...
splunk add search-server -host : -auth : -remoteUsername -remotePassword
When we tried to re-run the command we got an error about the cluster slave already existing. After a little more digging we realized that we need to connect to each of the search heads and assign the admin role that ability to see the indexes. We had done this on all of the indexers, but had not done it on the search heads.
You can also add search peers through Splunk Web on each search head. To do this, you must first unhide the hidden settings, as described in "The Settings menu." Then follow the instructions in "Add search peers to the search head."
We can see data on each of the search heads from the indexer. Now I just need to figure out:
1) How to replicate those role settings to all servers without having to manually touch them?
2) Why the DMC doesn't see the search heads (or license server for that matter!)?
Thanks for helping us find the correct path even if it wasn't the exact right answer.