topic Re: Moving Splunk to New Server - The WebServer doesn't start in Security

Moving Splunk to New Server - The WebServer doesn't start

rpearson — Tue, 23 Mar 2021 21:35:40 GMT

I have scoured the forums and checked the web_service.log but I can't seem to be able to figure out what my problem is. What can I be looking for?

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main nagios nix perfmon summary windows
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.1.2-545206cc9f70-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8000 to be available..

Re: Moving Splunk to New Server - The WebServer doesn't start

96nick — Wed, 24 Mar 2021 18:52:35 GMT

A couple questions:

I'm assuming that it's getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available.." but I don't want to assume. Is that correct?
What were you working on before you (re)started the web server? SSL/TLS?
What user is running Splunk? (root? splunk?) Have you tried doing a chown -R splunk:splunk {SPLUNK DIR}?

Re: Moving Splunk to New Server - The WebServer doesn't start

rpearson — Wed, 24 Mar 2021 19:03:06 GMT

1. Yes

2. I copied over my entire splunk directory over to a new server and installed splunk as per the instructions I am trying to turn it on for the new server for the first time.

3. I did chown the directory.

Re: Moving Splunk to New Server - The WebServer doesn't start

Vardhan — Wed, 24 Mar 2021 19:11:18 GMT

Hi @rpearson ,

Can you go to /opt/splunk/etc/system/local

Take a backup and remove the inputs.conf & server. conf. After that restart the Splunk and check.

Re: Moving Splunk to New Server - The WebServer doesn't start

rpearson — Wed, 24 Mar 2021 19:12:07 GMT

No change after removing those files.

Re: Moving Splunk to New Server - The WebServer doesn't start

rpearson — Wed, 24 Mar 2021 19:13:19 GMT

1. Yes

2. I copied over my entire splunk directory over to a new server and installed splunk as per the instructions I am trying to turn it on for the new server for the first time.

3. I did chown the directory.

@96nick

Re: Moving Splunk to New Server - The WebServer doesn't start

96nick — Wed, 24 Mar 2021 19:28:44 GMT

Have you checked out Splunkd.log on the new server? I'd grep for ERROR and see if it's telling you anything useful.

Are the Splunk versions the same old=>new? Things can get funky if you're going from 7.x to 8.x+ with the changes that Splunk put out (namely Python). If so try removing your apps (just move them to your home dir for now) and seeing if Splunk restarts.

Another thing to check is if something is up with your conf files you copied over in /etc/system/local. You may need to enter in the passwords in plaintext so they can be hashed again by Splunk itself.