when I do a ./splunk restart it seems to stop and start normally accept that the web server seems to take a long time to load.
the ............... while waiting for web server takes more that one line to complete and the web server is almost always up before the splunk startup script knows it is up.
What can I check to see why Splunk web is taking so long to respond to the start command?
I think this is not an issue to be worried of, I faced the same scenario a few times. If you are on a Linux Machine try doing a
tail -f /opt/splunk/var/log/splunk/splunkd.log while you perform
./splunk restart to restart the Splunk daemon. In some way, splunkd tries to access https://www.splunk.com/ or tries to resolve the name.
The issue you are facing will be seen, If the Machine where Splunk Is installed does not have connectivity to public internet i.e. Splunk web cannot resolve/connect to splunk.com. So it keeps trying to resolve the name but fails and Finally after a huge waiting Splunk Web starts. You can test this by Providing Connectivity to the instance, Splunk web will come up within a few seconds, once you disable the connectivity the problem will reappear.
So this is not an issue to be worried, it's just the lack of connectivity Splunk is trying to. it tries to resolve and fails.
Check the ownership/permissions on all the files under your Splunk home directory. If necessary, reset the file ownership. I had a similar problem once - and resetting the file ownership fixed it. (I had carelessly edited a configuration file and changed its ownership.) Of course, your problem could be completely different.
There must be an underlying issue. I am facing the same problem, made sure the permissions of the splunk directory were default (root user installation). Would it make a difference if the environment is SHC?