Moving Splunk to New Server - The WebServer doesn't start

[rpearson]

I have scoured the forums and checked the web_service.log but I can't seem to be able to figure out what my problem is.  What can I be looking for?

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main nagios nix perfmon summary windows
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.1.2-545206cc9f70-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8000 to be available..

[rpearson]

1.  Yes

2.  I copied over my entire splunk directory over to a new server and installed splunk as per the instructions I am trying to turn it on for the new server for the first time.

3.  I did chown the directory.

[Vardhan]

Hi @rpearson ,

Can you go to /opt/splunk/etc/system/local

Take a backup and remove the inputs.conf & server. conf. After that restart the Splunk and check.

[rpearson]

No change after removing those files.

[96nick]

A couple questions:

1. I'm assuming that it's getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available.." but I don't want to assume. Is that correct?
2. What were you working on before you (re)started the web server? SSL/TLS? 
3. What user is running Splunk? (root? splunk?) Have you tried doing a chown -R splunk:splunk {SPLUNK DIR}?

[rpearson]

1.  Yes

2.  I copied over my entire splunk directory over to a new server and installed splunk as per the instructions I am trying to turn it on for the new server for the first time.

3.  I did chown the directory.

@96nick 

[96nick]

Have you checked out Splunkd.log on the new server? I'd grep for ERROR and see if it's telling you anything useful. 

Are the Splunk versions the same old=>new? Things can get funky if you're going from 7.x to 8.x+ with the changes that Splunk put out (namely Python). If so try removing your apps (just move them to your home dir for now) and seeing if Splunk restarts. 

Another thing to check is if something is up with your conf files you copied over in /etc/system/local. You may need to enter in the passwords in plaintext so they can be hashed again by Splunk itself.