ERROR CMSlave - Waiting for the cluster manager to come up... (retrying every second)

[KSV]

hi guys
After adding [clustering] stanza and [replication_port://9887] in indexer cluster,  getting the below error
ERROR  - Waiting for the cluster manager to come up... (retrying every second),but  service is running, but it stuck at this point - Waiting for web server at http://<ip>:8000 to be available.
later, got this warning, WARNING: web interface does not seem to be available!
how to I fix this issue ?
8000,9887 ports are already open, I've passed same pass4SymmKey for all 3 servers in indexer cluster
tags: @any
 

[KSV]

hi @isoutamo 
yes, this is on indexer cluster - on all 3 nodes ID1,ID2,ID3
yes, master node is up and running, I can see no error logs from splunkd.log from master

[isoutamo]

Have this cluster work earlier or are you just putting it up from scratch?
Did you see any connections attemps in master's internal logs?
Are there any other messages in peer's internal logs?
Can you use e.g. curl or nc to connect masters management port from peer?

[KSV]

I'm creating it from scratch, 8080,9887 is not in listen state.
i'm unable to connect to master from peer

ProxyConfig  - Failed to initialize http_proxy from server.conf for splunkd. P
lease make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
INFO ProxyConfig  - Failed to initialize https_proxy from server.conf for splunkd.
Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
INFO ProxyConfig - Failed to initialize the proxy_rules setting from server.conf f
or splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled.
INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for
splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
INFO WatchedFile  - File too small to check seekcrc, probably truncated. Will re-re
ad entire file='/opt/splunk/var/log/introspection/http_event_collector_metrics.log'.
WARN SSLOptions  - server.conf/[search_state]/sslVerifyServerCert is false disab
ling certificate validation; must be set to "true" for increased security
 WARN SSLOptions- <internal>.conf/[<internal>]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
 IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s

[isoutamo]

Those messages are quite normal and not describe what issues you have.

Have you try e.g. nc or curl to check, if master is listening peers and response anything? Is pass4symKey working or are there any messages for it in _internal? btw when you post logs, please use block element </> where you paste those lines. It's much easier to read and we can be sure that those are what you have pasted.

If the connection between master and peer is working there are lot of messages in _internal.

[isoutamo]

Hi
Was this on indexer?
Have you checked that master node is up and running?
If it's up and running, there should be some more hints on its internal logs. Just check those to get more hints.
r. Ismo