Problem Starting Splunkweb

[raidercom]

I'm having problems getting Splunk to run on Ubuntu Server.

I've followed the installation instructions here for the debian installer:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/InstallonLinux
And followed the first run instructions here:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/StartSplunkforthefirsttime

When starting splunk, the start process hangs for about 20 seconds before displaying Starting splunkweb... Done.
If I do sudo /opt/splunk/bin/splunk status right away, splunkweb shows as running.
If I wait a couple of minutes, and run the same command, splunkweb shows as stopped. The wierd thing is that even though splunk status says that splunkweb is not running, netstat shows that it is listening, and ps shows that the process is python.

When I try to connect to the webpage, it will load after about 1.5 min loading time. When I try and do the first logon as admin, it eventually brings a page with the following error:

503 Service Unavailable

Return to Splunk home page
The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

I've tried uninstalling splunk completely and installing it again
Before running splunk, I've checked and the ports it uses are not in use.

I have the web_service.log file, but I cannot make any sense of it.

Ubuntu Server 11.10 x86

A little more about the environment:
The Linux box is a router that serves as a gateway for my network.
It hosts DNS (now),
DHCP, SSH
Apache, MySQL (internlly accessible-only website)

Trying to get Splunk operational on it, so that I pull logs from the Linux server for easier readability as well as from my Windows machines on the network.

Thank you for any help you could provide.
-Jeff

[wallacd2018]

Make sure the license group is right. In other words make sure you have the server licensed. 
Also to search the errors . 

index=_internal host=<"affectedhost">   sourcetype="splunk_web_service" source="splunkhome/var/log/splunk/web_service.log"

[Drainy · Accepted solution]

Hello!
It sounds like like your machine may be blocking access to port 8089 which Splunkd requires. Do you have a firewall running? Ensure that 8089 and 8000 are open on it, I have experienced issues with this before.

Also SELinux can cause problems, as a quick test disable it (if you have it enabled), restart and try again. If it is the cause and you do want to use SELinux then the SELinux IRC channel might be a good shout as they are pretty great at helping people configure up services for that sort of behavior.

[raidercom]

That did it. I don't know why but even with everything running on the same system, and talking via localhost, it required outbound firewall access on port 8089. I previously had all of the other ports allowed, just not that one.
Thanks!
-Jeff

[bojanz]

Check your DNS settings.
This looks as Splunk cannot resolve itself so it hangs when the web service is being started.
Splunk needs a working DNS server - it does not need to be able to resolve everything (so an NXDOMAIN answer is OK), but the DNS server must be reachable.

[raidercom]

I just setup a DNS server and clients and the Linux box itself can lookup the hostname/IP of the Linux box that is my Splunk server.

I am still getting the same error, splunk web shows as running for a while (but I cannot get it to do anything), then stops, but it is still listening on the port 8000

-Jeff

[raidercom]

I only have an external DNS server setup.
I'll start working on getting a local DNS server setup.
Thanks
-Jeff

[Ayn]

Does the system have properly configured DNS settings?