How to troubleshoot splunk web server not starting?

[maweyandt]

Hello, I'm having an odd issue that started after I went to restart. Issuing the ./splunk start command as the relevant user splunkd starts right up, but the start hangs at...

'Waiting for web server at https://127.0.0.1:8000 to be available.....'

Ports are open, server is listening on 8000 and my netstat -an | grep 8000 gives me

tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:57576 127.0.0.1:8000 ESTABLISHED
tcp 308 0 127.0.0.1:8000 127.0.0.1:57576 ESTABLISHED

splunk command owns the process using port 57576 and splunkd is the listen/port 8000.

Not getting any errors in splunkd.log or web_service.log.

How should I proceed?

[javiergn]

Depending on the Linux flavour you are using and the packages installed, you might need to disable your local firewall, either iptables or firewall. In my case, because of the security restrictions, our CentOS 7 servers have the firewall enabled by default on any non-restricted port so all I had to do was:

# Web
firewall-cmd --zone=public --add-port=8000/tcp --permanent
# Management / REST API
firewall-cmd --zone=public --add-port=8089/tcp --permanent
# KV store
firewall-cmd --zone=public --add-port=8191/tcp --permanent
# Data
firewall-cmd --zone=public --add-port=9997/tcp --permanent
# Replication
firewall-cmd --zone=public --add-port=8080/tcp --permanent

firewall-cmd --reload

[maweyandt]

I've narrowed it down to a permission issue (webserver will start up when initiated as root) but all files are owned by my splunk user in the /opt/splunk directory so I'm not sure why it's failing under the splunk user. Any suggestions would be great! Thanks!

[rvencu]

got the same problem after configuring as a service under systemd

if I use root:root as user for the service it works, even if the splunk home is owned by splunk user

but I am not sure if this is ok to use, I would prefer it to work under splunk user as it should

If you found a solution please share

[PickleRick]

No, you should not run services as root if you can avoid it (and in splunk's case it's generally not needed).

In case of splunk failing to start under a normal user, it might be a SELinux issue. What's your SELinux enforce status?

[dkwilcox]

I have the same issue.  But starting splunk with root doesn't work.  Firewall is not running. I got a new Certificate and created a new PEM files.  No errors in the logs.  Just keep waiting on the startup of the HTTPS.

Any suggestions?

[bgagliardi1]

Hey Maweyandt, I've ran into an array of issues with the splunk web server not showing up myself. Some tips I'd recommend:

1) Always run splunk as the splunk user
2) Make sure splunk is the owner of all files in Splunk home. (usually /opt/splunk
3) If you're running an upgrade and it won't come up. Make a copy of all of your apps, remove them, and try starting it. If it starts - you know you have an app that is a culprit and you'll have to add/remove 1 by 1 until you find the culprit.