Deployment Architecture
Highlighted

How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

Path Finder

Hi:

I am unable to start splunk web on version 6.2.2 on RHEL 6.5.

Checking prerequisites...
        Checking http port [8001]: open
        Checking mgmt port [8089]: open
The Splunk web interface is at https://awspoclts21:8001

I disabled the local firewall, but in vain. Still unable to start Splunk Web. Is there something I am missing. There is nothing on the web_services.log

015-04-07 11:12:31,979 INFO    [5523f3df637fb21415ec90] root:635 - CONFIG: version_label (str): 6.2.2
2015-04-07 11:12:31,979 INFO    [5523f3df637fb21415ec90] root:635 - CONFIG: version_number (str): 6.2.2
2015-04-07 11:12:31,979 INFO    [5523f3df637fb21415ec90] root:635 - CONFIG: x_frame_options_sameorigin (bool): True
2015-04-07 11:12:31,980 INFO    [5523f3df637fb21415ec90] root:693 - DJANGO: configuring...
2015-04-07 11:12:32,122 INFO    [5523f3df637fb21415ec90] root:735 - DJANGO: not starting, found no apps
2015-04-07 11:12:32,122 INFO    [5523f3df637fb21415ec90] root:129 - ENGINE: Bus STARTING
2015-04-07 11:12:32,132 INFO    [5523f3df637fb21415ec90] root:129 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2015-04-07 11:12:32,239 INFO    [5523f3df637fb21415ec90] root:129 - ENGINE: Serving on 127.0.0.1:8065
2015-04-07 11:12:32,240 INFO    [5523f3df637fb21415ec90] root:129 - ENGINE: Bus STARTED
2015-04-07 11:12:32,286 INFO    [5523f3e0487fb211aa1210] root:129 - ENGINE: Started monitor thread 'Monitor'.

splunkd.log

04-07-2015 11:13:01.406 -0400 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
04-07-2015 11:13:01.406 -0400 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
04-07-2015 11:13:01.406 -0400 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
04-07-2015 11:13:21.124 -0400 WARN  TcpOutputProc - Cooked connection to ip=10.143.28.20:9997 timed out
04-07-2015 11:13:43.126 -0400 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.143.29.20_8089_awspoclts21.rsc.humad.com_awspoclts21_0A653A89-0248-4601-A642-A716953F5A43
04-07-2015 11:13:43.225 -0400 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.143.29.20_8089_awspoclts21.rsc.humad.com_awspoclts21_0A653A89-0248-4601-A642-A716953F5A43
04-07-2015 11:13:51.125 -0400 WARN  TcpOutputProc - Cooked connection to ip=10.143.28.20:9997 timed out
04-07-2015 11:14:21.127 -0400 WARN  TcpOutputProc - Cooked connection to ip=10.143.28.20:9997 timed out
04-07-2015 11:14:43.252 -0400 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.143.29.20_8089_awspoclts21.rsc.humad.com_awspoclts21_0A653A89-0248-4601-A642-A716953F5A43
04-07-2015 11:14:51.128 -0400 WARN  TcpOutputProc - Cooked connection to ip=10.143.28.20:9997 timed out
04-07-2015 11:15:21.130 -0400 WARN  TcpOutputProc - Cooked connection to ip=10.143.28.20:9997 timed out

We can ignore the errors on splunkd.log as this is a fresh install and I am planning to add port 9997 through the web UI

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

Splunk Employee
Splunk Employee

As per your first bits it is started... do you mean you can't access it?

I guess I have to ask... it looks like you changed the default port for the web interface. And you're sure you're going to https://awspoclts21:8001 and not https://awspoclts21:8000 out of habit? And you tried using the IP address rather than the hostname, yes?

Also... you've edited the startup feedback quite a bit. You might want to post the whole thing... Just in case there is something odd and easy to miss.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

Motivator

to start Splunk Web, go to browser and run:

http://localhost:8001
if you wan u can run splunkd to do it go to Splunk_home/bin directory and run splunk start command
before run in browser:
http://localhost:8001
or
127.0.0.1:8001

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

Path Finder

Yes I am going to awspoclts21:8001. I see using netstat command that the port 8001 is listening. But when I am doing a telnet from my local host pc onto awspoclts21 on port 8000 it is stating connection refused. I spoke with the network firewall team and they also suggest they don't have any firewall rules set on port 8000. I need to check on the server side.

On the server side I have disabled the firewall services, using service iptables stop.

Could you suggest how do I proceed next?

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

SplunkTrust
SplunkTrust

You have configured splunkweb to run on port 8001, then why are you checking port 8000 on firewall? Telnet from your machine to awspoclts21 on port 8001

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

Path Finder

I have changed the port to 8001 and telnet doesn't work. First tried changing web port from 8000 to 8001 and did telnet from my machine to awspoclts21 on port 8001. That doesn't work.

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

SplunkTrust
SplunkTrust

Then port 8000 & 8001 are blocked from your local machine to awspoclts21. Check with your network admin that any firewall rule is blocking this traffic.

0 Karma
Highlighted

Re: How to troubleshoot why I am unable to start Splunk Web using version 6.2.2 on RHEL 6.5?

New Member

Try accessing localhost:8001, localhost:8002. I had similar issue and starting the splunk process clarified the logs. Which started the web UI on port 8002
,Try accessing localhost:8001, localhost:8002.
I had similar issue and the splunk start process clarified the web UI was started on port 8002.

0 Karma