Why am I unable to bootstrap a captain in a search head cluster?

[singhbc]

When i send in the the command

./splunk bootstrap shcluster-captain -servers_list “https://10.100.97.116:8089,https://10.100.97.117:8089,https://10.100.97.118:8089" -auth admin:<password>

I get the following response on the command line

">"

and nothing happens subsequently and I have to ctrl^c to get back to command prompt.

./splunk show shcluster-status produces the following result

In handler 'shclusterstatus': This node is not the captain of the search head pool, and we could not determine the current captain. The pool is either in the process of electing a new captain, or this member hasn't joined the pool.

[jreuter_splunk · Accepted solution]

Do you have any special characters in your password that could be interpreted by the shell?

localguy@localhost.localdomain:/home/> splunk search -auth admin:somePasswordWithAQuotation"MarkInIt
>
> ^C

Try using single quotes:

localguy@localhost.localdomain:/home/> splunk search -auth 'admin:somePasswordWithAQuotation"MarkInIt'
Login failed

[jreuter_splunk]

Are the double quotes cut and pasted? They look like fancy smart quotes to me, but I am not sure if that is reformatting. You might try re-entering them manually. The '>' is an indication that the shell is awaiting further input, and one smart quote would have the same effect:

> splunk search “index=_internal earliest=-1m@m" -auth 'admin:admin@splunk.com'
>
> ^C

[singhbc]

You did it! It was the silly double quotes, they were different on the left side and the correct ones on the right side, that was causing it.
Thanks a LOT!.

[jreuter_splunk]

They are pretty sneaky, and have bitten me more than once. I didn't even notice them on the first pass!

[singhbc]

Do have an @ sign in the password but single quotes didnt help as well.

[esix_splunk]

Did you build these from clean instances? Or existing? Are all nodes up and running while you try to boostrap?

Can you show the [shclustering] stanza from your server.conf also.

[esix_splunk]

What replication port did you configure, and is it open between all servers?

DEV license should allow you to run SHC, but Im not sure about multisite clustering. These are two different features.

[singhbc]

[replication_port://8901] for 116
[replication_port://8902] for 117
[replication_port://8903] for 118

When i restart splunk on CLI on each of them it does check and report that the replication ports are open.
How do I check that they are all open between all servers?

[esix_splunk]

Your replication port needs to be the same for all nodes. Correct that in the server.conf, and try again.

[singhbc]

Any other troubleshooting suggestions?

[singhbc]

I changed them to 8901 for all three SH Nodes.
Restarted them but I still get the same ">" outcome.

[singhbc]

i can telnet to the the replication ports from each node to another node. Therefore the ports are open as well as communicating. Any more suggestions?

[singhbc]

Does License have to do anything with this behavior?

My license is a "developer license" and it says at the bottom of the list of enabled features

• UnisiteClustering

and I have enabled multisite clustering for this index cluster even though the SH cluster is fully on site1.

[singhbc]

shclustering
conf_deploy_fetch_url = https://10.100.97.115:8089
disabled = 0
mgmt_uri = https://10.100.97.116:8089
pass4SymmKey = $1$phcpQF+xcl8+
replication_factor = 2

shclustering
pass4SymmKey = $1$/Age1HwLQV0Q

All the three server.conf have the same stanza except the mgmt_uri which is 117 and 118.
Yes I started from clean instances. All nodes are up and running, I can see them with UP status on the Master settings page.