Deployment Architecture

Why am I unable to bootstrap a captain in a search head cluster?

singhbc
Path Finder

When i send in the the command

./splunk bootstrap shcluster-captain -servers_list “https://10.100.97.116:8089,https://10.100.97.117:8089,https://10.100.97.118:8089" -auth admin:<password>

I get the following response on the command line

">"

and nothing happens subsequently and I have to ctrl^c to get back to command prompt.

./splunk show shcluster-status produces the following result

In handler 'shclusterstatus': This node is not the captain of the search head pool, and we could not determine the current captain. The pool is either in the process of electing a new captain, or this member hasn't joined the pool.
0 Karma
1 Solution

jreuter_splunk
Splunk Employee
Splunk Employee

Do you have any special characters in your password that could be interpreted by the shell?

localguy@localhost.localdomain:/home/> splunk search -auth admin:somePasswordWithAQuotation"MarkInIt
>
> ^C

Try using single quotes:

localguy@localhost.localdomain:/home/> splunk search -auth 'admin:somePasswordWithAQuotation"MarkInIt'
Login failed

View solution in original post

jreuter_splunk
Splunk Employee
Splunk Employee

Do you have any special characters in your password that could be interpreted by the shell?

localguy@localhost.localdomain:/home/> splunk search -auth admin:somePasswordWithAQuotation"MarkInIt
>
> ^C

Try using single quotes:

localguy@localhost.localdomain:/home/> splunk search -auth 'admin:somePasswordWithAQuotation"MarkInIt'
Login failed

jreuter_splunk
Splunk Employee
Splunk Employee

Are the double quotes cut and pasted? They look like fancy smart quotes to me, but I am not sure if that is reformatting. You might try re-entering them manually. The '>' is an indication that the shell is awaiting further input, and one smart quote would have the same effect:

> splunk search “index=_internal earliest=-1m@m" -auth 'admin:admin@splunk.com'
>
> ^C

singhbc
Path Finder

You did it! It was the silly double quotes, they were different on the left side and the correct ones on the right side, that was causing it.
Thanks a LOT!.

jreuter_splunk
Splunk Employee
Splunk Employee

They are pretty sneaky, and have bitten me more than once. I didn't even notice them on the first pass!

0 Karma

singhbc
Path Finder

Do have an @ sign in the password but single quotes didnt help as well.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Did you build these from clean instances? Or existing? Are all nodes up and running while you try to boostrap?

Can you show the [shclustering] stanza from your server.conf also.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What replication port did you configure, and is it open between all servers?

DEV license should allow you to run SHC, but Im not sure about multisite clustering. These are two different features.

0 Karma

singhbc
Path Finder

[replication_port://8901] for 116
[replication_port://8902] for 117
[replication_port://8903] for 118

When i restart splunk on CLI on each of them it does check and report that the replication ports are open.
How do I check that they are all open between all servers?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Your replication port needs to be the same for all nodes. Correct that in the server.conf, and try again.

singhbc
Path Finder

Any other troubleshooting suggestions?

0 Karma

singhbc
Path Finder

I changed them to 8901 for all three SH Nodes.
Restarted them but I still get the same ">" outcome.

0 Karma

singhbc
Path Finder

i can telnet to the the replication ports from each node to another node. Therefore the ports are open as well as communicating. Any more suggestions?

0 Karma

singhbc
Path Finder

Does License have to do anything with this behavior?

My license is a "developer license" and it says at the bottom of the list of enabled features

  • UnisiteClustering

and I have enabled multisite clustering for this index cluster even though the SH cluster is fully on site1.

0 Karma

singhbc
Path Finder

shclustering
conf_deploy_fetch_url = https://10.100.97.115:8089
disabled = 0
mgmt_uri = https://10.100.97.116:8089
pass4SymmKey = $1$phcpQF+xcl8+
replication_factor = 2

shclustering
pass4SymmKey = $1$/Age1HwLQV0Q

All the three server.conf have the same stanza except the mgmt_uri which is 117 and 118.
Yes I started from clean instances. All nodes are up and running, I can see them with UP status on the Master settings page.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...