Web server on Splunk cluster master does'nt starts , below is the mesage which i see when starting it
Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
Done
Bypassing local license checks since this instance is configured with a remote license master.
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.0.5-a1a6394cc5ae-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
[ OK ]
Waiting for web server at https://127.0.0.1:8443 to be available.
One thing which i see in my web_Service logs are
root:730 - CONFIG: error_page.default (method): <bound method ErrorController.handle_error of <splunk.appserver.mrsparkle.controllers.error.ErrorController object at 0x7fb7fdef6e10>>
Other thing seems ok.
From splunkd log file i have error messages related to my remote license server:
ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=XXX
This is intentially kept like that because i want my server's to run without a license server.These are test servers and i don't want them to connect to a license server.
Another thing is an warning related to searchead:
WARN DistributedPeer - Peer:https://searchhead.splunk.test:8089 Failed to get server info from https://searchhead.splunk.test:8089/services/server/info response code=401
I tried to telnet from my searchead to cluster master on port 8089 and the connectivity looks ok.
I can upload the complete files if required.
there is no error in the log. can't guess whats happening .