Installation

Can't access splunk web server

dataIngester
Explorer

Hi Splunk Community,

I know this question has been asked a several times over. But I don't find a desirable solution to this.

We have been installing Splunk Enterprise on various virtual servers each for a Search Head, Indexer, HF. So far we have installed more than 5 Splunk Enterprise on each Linux (RHEL) VM, following the standard installation procedure. Also keeping the splunk.secret file the same throughout. Every server is functioning normally. Except for on one server, we cannot access the splunk web interface via localhost.

Note: None of the configurations have been changed.

the web.conf file has the 

 

startwebserver = 1

 

and 

 

httpport = 8000

 

 

The netstat -an | grep 8000 shows that it is listening on this port

 

tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN

 

 

I have checked if its the firewall issue, with performing a telnet 127.0.0.1 8000, as well as telnet 0.0.0.0 8000 and it gets connected

 

Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

 

So I believe that there is no firewall issue as well.

The only valuable info I receive on using the "Escape character" after the telnet connection is

 

HTTP/1.1 400 Bad Request
Date: Mon, 12 Oct 2020 16:44:47 GMT
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 207
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>400 Bad Request</title></head><body><h1>Bad Request</h1><p>HTTP Request was malformed.</p></body></html>
Connection closed by foreign host.

 

 

I can't seem to find anything on the splunkd.log too.

I'm running out of options on how to debug this issue as to why the web server is not loading. Please assist.

 

Thanks,

Labels (4)
0 Karma
1 Solution

dataIngester
Explorer

Thanks a lot @Richfez for all the help. It turns out it was indeed a FW issue. Our Infra team had not opened up the FW Rule for the port 8000.

Thanks once again for your replies and support. Appreciate it!

Best Regards,

dataIngester

View solution in original post

0 Karma

Richfez
SplunkTrust
SplunkTrust

That Bad Request is exactly what you should get - an "escape character" isn't a valid HTML request.  I just tested that my perfectly fine, working server gives exactly that same response.  And in fact, that response sort of indicates the web server IS running, and working correctly enough to identify that it was an invalid request and to send back and error message to your telnet session.  So this is good.

I suspect the problem is actually your web browser or something higher up the stack.

I see no mention of trying a web browser from another machine pointed to this machine's IP address and port 8000.  Could you try that please and let us know?

http://10.1.2.3:8000 or whatever.

I suspect you'll find a massive facepalm at some point, but for now if it's working from actual client workstations, is there actually any problem?

 

Speaking of which, I have to admit it's been a decade or more since I had a full windowing system on a linux server to even *try* using the "local web browser" to browse a local "web site".   Why are you doing full desktop installs?  Probably all sorts of weird things that could go wrong in that. 🙂

 

 

 

0 Karma

dataIngester
Explorer

@Richfez : It's good to know that the web server is running correctly for this instance. 

Yes, indeed the 4xx HTML response codes are generally something to do with the web clients. I tried using Internet Explorer as well (initially on Chrome), but with no luck.

Apologies, the earlier info I provided was a little misleading, it wasn't via a localhost. Basically to give you an understanding of the infrastructure set up, Splunk is installed on virtual Linux Server (hosted elsewhere) and the client is a Windows VDI (of course hosted elsewhere). And we connect to these Splunk Servers (on a Linux VM) via the Windows VDI client using the hostname of the server to access the web UI. Yes, I did try connecting to that server using its http://<IP_address>:<PORT> , but again with no success.

Any ideas on how to debug the issue and get the web UI working? 

0 Karma

Richfez
SplunkTrust
SplunkTrust

Oh wow, that's very interesting then.  I hadn't heard of an issue like this.

Now that you've clarified it a bit, I'm not quite as positive the web server IS actually running.  I mean, probably, but just that I am not longer as sure of it.  Still, my previous answer may be useful to other folks, so we'll leave it.  Just don't mark it as accepted!

Have you opened a support ticket?  That's what I'd be doing, even if in tandem with this process here.

Try checking

index=_internal sourcetype=splunk_web_service (engine OR proxied)

Or even

index=_internal sourcetype=splunk_web_service (ERROR OR WARN)

If those don't show much, drop even the ERROR OR WARN off it,restart and look at the whole set of splunk_web_service logs.

I also tried to find a logging setting for any of the web service so we could crank it up to debug, but a quick search shows it doesn't have it's own, well named logging facility.  Hrmmm.

 

Secondarily, maybe look at areas like

index=_internal sourcetype=splunk_web_access

and see what it does when you try to access a web page for it and get that error.

So to recap,

1) Open a support ticket

2) Check those logs and see if they lead anywhere useful or offer any hints.

-Rich

 

dataIngester
Explorer

Thanks a lot @Richfez for all the help. It turns out it was indeed a FW issue. Our Infra team had not opened up the FW Rule for the port 8000.

Thanks once again for your replies and support. Appreciate it!

Best Regards,

dataIngester

View solution in original post

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!