How to troubleshoot why I can no longer access Splunk Web on a Linux server?

[ajchz]

Hello All,

I have checked the forums and I have tried several resolutions, but none seems to work. Yesterday I installed Splunk on a LINUX local server and I was able to get the web interface but today I cannot.

I'm able to get the Splunk Atom Feed, and I can play with it.

What I have tried so far:
I tried changing the port from 8000 to 8081 since NGINX was using the port (according to the Splunk Errors Log).

sudo /opt/splunk/bin/splunk status
Output:

splunkd is running (PID: 3454).
splunk helpers are running (PIDs: 3455 3488).

sudo /opt/splunk/bin/splunk restart
Output:

Stopping splunkd... Shutting down.  Please wait, as this may take a few minutes.

Stopping splunk helpers...

Done.

Splunk> Australian for grep.

Checking prerequisites...   Checking mgmt port [8089]: open     Checking configuration...  Done.    Checking critical directories...    Done    Checking indexes...         Validated:
_blocksignature _thefishbucket nginx sample     Done    Checking filesystem compatibility...  Done  Checking conf files for problems...     Done All preliminary checks passed.

Starting splunk server daemon (splunkd)...   Done

sudo netstat -an
No issue with port 8081

So far I'm not able to get it work, is reinstalling a solution?

Thanks in advance for your help.

[ajchz]

So far, this is my solution: I reinstalled Splunk again and seems to be working fine.
I restarted the machine and it works with no issue, I can access Splunk.

I'm going to check during the coming 7 days and report any error or if reinstalling was the solution.

[grijhwani]

From the linux machine, telnet to the port in question. Do you get a connection? If so, the problem is the network and/or intervening firewalls.

What do you see with tcpdump on the Splunk machine? Do the query packets reach it? Can you tcpdump on the client (browser machine). If so, can you see the packets going out and responses returning as expected (including to the original SYN, SYN/ACK exchange when the connection is opened?)

You say the port is not being blocked. Where is it not being blocked? (I know - seems an absurd question, but it's not really.) Are you and the Splunk machine on the same network or is there a firewall/filtered routing in between? Have you checked for blocking all points in between.

[ajchz]

Hello,

I telnet the port and it got a connection. So, I reinstalled Splunk again and seems to be working fine.
I restarted the machine and it works with no issue, I can access Splunk.

I'm going to check during the coming 7 days and report any error or if reinstalling was the solution.

[esix_splunk]

Check and make sure your firewall is allowing to traffic TCP/8001. Additionally, run btool and validate your configuration is accurate:

/opt/splunk/bin/splunk btool web list --debug | grep httpport
/opt/splunk/etc/apps/local/web.conf httpport = 8000

[ajchz]

Hello thanks for replying. I got the following:
sudo /opt/splunk/bin/splunk btool web list --debug | grep httpport
/opt/splunk/etc/system/default/web.conf httpport = 8081

I can confirm port is not being blocked