Splunk Search
Highlighted

Clustered search heads not seeing data from clustered indexers

Path Finder

We are building a single-site pilot environment with the following layout:

1 x Deployment and License Manager
3 x Search heads (configured in a SH cluster)
3 x Indexers
1 x Indexer Cluster Master

We have the indexers set up and sharing data and we even have a test forwarder sending data to an index on those clusters, however, when we connect to the search heads (either directly or via our load-balanced IP in front of them) the search does not see any data whether from the internal indexes or the external test index we built for our test data.

We ran the /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list command and listed all of our search heads. We also ran the /opt/splunk/bin/splunk edit cluster-config-mode searchhead -master_uri on the captain.

We've read through the http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/SHCandindexerclusterdocumentation but we're not clear whether we need to run edit cluster-config -mode searchhead -master_uri command on each of the search heads. That is what we've done, but it doesn't appear to resolve the issue.

0 Karma
Highlighted

Re: Clustered search heads not seeing data from clustered indexers

Path Finder

Here is the output of the /opt/splunk/bin/splunk list cluster-config command:

    config
            access_logging_for_heartbeats:1
            cxn_timeout:60
            disabled:0
            forwarderdata_rcv_port:?
            forwarderdata_use_ssl:0
            heartbeat_period:0
            heartbeat_timeout:60
            master_uri:https://[IP of the index cluster master]:8089
            max_peer_build_load:5
            max_peer_rep_load:5
            mode:searchhead
            multisite:false
            percent_peers_to_restart:10
            ping_flag:1
            quiet_period:60
            rcv_timeout:60
            rep_cxn_timeout:60
            rep_max_rcv_timeout:600
            rep_max_send_timeout:600
            rep_rcv_timeout:60
            rep_send_timeout:60
            replication_factor:3
            replication_use_ssl:0
            restart_timeout:60
            search_factor:1
            search_files_retry_timeout:600
            send_timeout:60
            site:default
0 Karma
Highlighted

Re: Clustered search heads not seeing data from clustered indexers

Path Finder

Also enabled the DHC and confirmed that we are not seeing a search head cluster defined there either.

0 Karma
Highlighted

Re: Clustered search heads not seeing data from clustered indexers

SplunkTrust
SplunkTrust

Hi joshuabiggley, You will want to setup each index cluster slave as a search peer on each search head cluster member. I believe this will resolve your issue. Please let me know how it works for you! 😄

View solution in original post

0 Karma
Highlighted

Re: Clustered search heads not seeing data from clustered indexers

Path Finder

We had already added the cluster slaves as search peers on each search head cluster member. We used the command below...

splunk add search-server -host : -auth : -remoteUsername  -remotePassword 

When we tried to re-run the command we got an error about the cluster slave already existing. After a little more digging we realized that we need to connect to each of the search heads and assign the admin role that ability to see the indexes. We had done this on all of the indexers, but had not done it on the search heads.

http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/Connectclustersearchheadstosearchpeers

You can also add search peers through Splunk Web on each search head. To do this, you must first unhide the hidden settings, as described in "The Settings menu." Then follow the instructions in "Add search peers to the search head."

We can see data on each of the search heads from the indexer. Now I just need to figure out:

1) How to replicate those role settings to all servers without having to manually touch them?
2) Why the DMC doesn't see the search heads (or license server for that matter!)?

Thanks for helping us find the correct path even if it wasn't the exact right answer.