Deployment Architecture

How to troubleshoot splunk web server not starting?

maweyandt
New Member

Hello, I'm having an odd issue that started after I went to restart. Issuing the ./splunk start command as the relevant user splunkd starts right up, but the start hangs at...

'Waiting for web server at https://127.0.0.1:8000 to be available.....'

Ports are open, server is listening on 8000 and my netstat -an | grep 8000 gives me

tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:57576 127.0.0.1:8000 ESTABLISHED
tcp 308 0 127.0.0.1:8000 127.0.0.1:57576 ESTABLISHED

splunk command owns the process using port 57576 and splunkd is the listen/port 8000.

Not getting any errors in splunkd.log or web_service.log.

How should I proceed?

Tags (1)
0 Karma

javiergn
SplunkTrust
SplunkTrust

Depending on the Linux flavour you are using and the packages installed, you might need to disable your local firewall, either iptables or firewall. In my case, because of the security restrictions, our CentOS 7 servers have the firewall enabled by default on any non-restricted port so all I had to do was:

# Web
firewall-cmd --zone=public --add-port=8000/tcp --permanent
# Management / REST API
firewall-cmd --zone=public --add-port=8089/tcp --permanent
# KV store
firewall-cmd --zone=public --add-port=8191/tcp --permanent
# Data
firewall-cmd --zone=public --add-port=9997/tcp --permanent
# Replication
firewall-cmd --zone=public --add-port=8080/tcp --permanent

firewall-cmd --reload
0 Karma

maweyandt
New Member

I've narrowed it down to a permission issue (webserver will start up when initiated as root) but all files are owned by my splunk user in the /opt/splunk directory so I'm not sure why it's failing under the splunk user. Any suggestions would be great! Thanks!

0 Karma

rvencu
Path Finder

got the same problem after configuring as a service under systemd

if I use root:root as user for the service it works, even if the splunk home is owned by splunk user

but I am not sure if this is ok to use, I would prefer it to work under splunk user as it should

If you found a solution please share

0 Karma

PickleRick
Ultra Champion

No, you should not run services as root if you can avoid it (and in splunk's case it's generally not needed).

In case of splunk failing to start under a normal user, it might be a SELinux issue. What's your SELinux enforce status?

0 Karma

dkwilcox
Observer

I have the same issue.  But starting splunk with root doesn't work.  Firewall is not running. I got a new Certificate and created a new PEM files.  No errors in the logs.  Just keep waiting on the startup of the HTTPS.

Any suggestions?

0 Karma

bgagliardi1
Path Finder

Hey Maweyandt, I've ran into an array of issues with the splunk web server not showing up myself. Some tips I'd recommend:

1) Always run splunk as the splunk user
2) Make sure splunk is the owner of all files in Splunk home. (usually /opt/splunk
3) If you're running an upgrade and it won't come up. Make a copy of all of your apps, remove them, and try starting it. If it starts - you know you have an app that is a culprit and you'll have to add/remove 1 by 1 until you find the culprit.

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...