Why are search head hosts unable to connect to master indexer?

[scottj1y]

I'm setting up some hosts to become a search head cluster to be joined with an indexer cluster. There are five hosts, one of which is going to be the Search Head cluster captain.

I've installed Splunk 6.5.2 on all of them and it's up and running with no errors. I'm now at the point of turning them into search heads and whether I use the CLI or from the Splunk Web UI, they all return the same error.

Could not contact master.  Check that the master is up, the master_uri=https://10.x.x.97:5500 and secret are specified correctly

But when I ran a netcat test on that port from each search head candidate to the master indexer and from the master indexer to each of the hosts it past every time

from each host to master

searchhead-001 Connection to 10.x.x.97 5500 port [tcp/fcp-addr-srvr1] succeeded!
searchhead-002 Connection to 10.x.x.97 5500 port [tcp/fcp-addr-srvr1] succeeded!
searchhead-004 Connection to 10.x.x.97 5500 port [tcp/fcp-addr-srvr1] succeeded!
searchhead-003 Connection to 10.x.x.97 5500 port [tcp/fcp-addr-srvr1] succeeded!


From Master to each host

Connection to 10.x.x.203 5500 port [tcp/fcp-addr-srvr1] succeeded!
Connection to 10.x.x.200 5500 port [tcp/fcp-addr-srvr1] succeeded!
Connection to 10.x.x.202 5500 port [tcp/fcp-addr-srvr1] succeeded!
Connection to 10.x.x.201 5500 port [tcp/fcp-addr-srvr1] succeeded!

When setting them up I've copied and pasted the URI and secret from a text file so I'm certain that I'm not mistyping either one.

The only one that has succeeded is the host that will be the search head captain and there's nothing different about it at all. I've even gone so far as to reimage each of the other hosts, reinstall RHEL 6.8.4, and reinstall and set up Splunk 6.5.2 but still am getting the same results.

What am I overlooking?

[lycollicott · Accepted solution]

Oh, in that case let's look at the command you used:

splunk edit cluster-config -mode searchhead -master_uri https://10.x.x.97:5500 -replication_port 8080 -secret <secret key -auth admin:<admin password>

If you're just trying to enable a simple single search head why have you included -replication_port 8080 ?

Try just this instead:

splunk edit cluster-config -mode searchhead -master_uri https://10.x.x.97:5500 -secret <secret key -auth admin:<admin password>

[scottj1y]

Out of the box this didn't work (meaning with the state the hosts were already in), but when I deleted the Splunk installation and then tried the command it succeeded. Thanks for the help.

[lycollicott]

Good stuff. I'm glad it's working for you.

[scottj1y]

Yes, I realized my initial post is not very clear. Although these hosts will eventually become a search head cluster, right now I'm merely trying to enable them as search heads and nothing else.

So the problem is, even as stand alone search heads I'm getting that error although one of the five hosts has no problem. All five hosts are otherwise identical in terms of set up and installed software. And, as you can see, there is no network connectivity problem either from the hosts to the master nor from the master to any of the hosts.

So I'm trying to figure out why they won't become search heads.

[lycollicott]

I think you need to first do a splunk init shcluster-config and then a splunk bootstrap shcluster-captain.

Look at this documentation for the details (http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/SHCdeploymentoverview) and I think it will help you out.

[lycollicott]

What did you run on the node to turn it into the captain? What is the command that is failing?

[scottj1y]

I haven't turned any node into anything as I wrote above. The only one that has succeeded is the host that will be the search captain.

I've only run the command to enable a host be a search head on any of them, i.e.,

splunk edit cluster-config -mode searchhead -master_uri https://10.x.x.97:5500 -replication_port 8080 -secret <secret key -auth admin:<admin password>