topic Re: My splunk instance stop responding in Deployment Architecture

My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 15:28:27 GMT

I have been running splunk on a server for sometime now and last week it stops responding. I don't know what could be the issue that cause it. When I access the server and go to localhost:8000, the page keeps loading and never give me any message. What can I do now?

Re: My splunk instance stop responding

akocak — Thu, 26 Jul 2018 15:32:36 GMT

Can you give more information about your OS and version of the Splunk ?

Re: My splunk instance stop responding

pradeepkumarg — Thu, 26 Jul 2018 15:32:39 GMT

Did you try restarting?
Any errors in splunkd.log? Is the server healthy? Try using servername:8000 or serverip:8000 and see if it makes any difference?

Re: My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 15:51:04 GMT

Hi I'm running version 6.6.2. And it is running on a windows server.

Re: My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 16:10:12 GMT

I did try to stopped, restarted it but even when it stop, my localhost:8000 still doesn't response. I would expect it to send me something like server's not responding. The server is healthy. There are 2 errors such as: ERROR HttpListener - Exception. I just tried using servername:8000 and serverip:8000 and it still doesn't work.

Re: My splunk instance stop responding

hortonew — Thu, 26 Jul 2018 17:06:24 GMT

Has it created a socket on port 8000 yet? Try: netstat -an | select-string 8000 and see if anything returns that it's listening. If not, do you have anything denying it from creating that socket? Anti-virus, etc?

Re: My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 18:31:12 GMT

I tried that command but it said select-string is not recognized as an internal or external command. I have McAfee running but I don't know if that blocking the port. How can I find out?

Re: My splunk instance stop responding

hortonew — Thu, 26 Jul 2018 18:33:57 GMT

Run command via powershell, not cmd.exe if that's what you were doing. select-string is a powershell command.

Re: My splunk instance stop responding

akocak — Thu, 26 Jul 2018 18:40:29 GMT

in CMD:
netstat -an | findstr /s /i 8000

Re: My splunk instance stop responding

woodcock — Thu, 26 Jul 2018 19:15:34 GMT

You probably ran out of disk space. What does df show you?

Re: My splunk instance stop responding

tamduong16 — Tue, 29 Sep 2020 20:39:46 GMT

I did that and it display a list of ip addresss along with a column to the right that indicate FIN_WAIT_2 and CLOSE_WAIT

Re: My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 19:46:29 GMT

I checked disk space and I still have at least 10 GB of disk space left.

Re: My splunk instance stop responding

hortonew — Thu, 26 Jul 2018 19:47:53 GMT

But nothing with LISTENING at the end? You should see an entry like:

tcp 0 0.0.0.0:8000 0.0.0.0:* LISTEN

So: netstat -an | select-string 8000 | select-string LIST

All the entries about fin/close are IPs trying to hit that socket and those connections being torn down.

Re: My splunk instance stop responding

tamduong16 — Tue, 29 Sep 2020 20:39:49 GMT

I tried to change the port to 9089 and now it showing: "This page isn't working" ERR_EMPTY_RESPONSE

Re: My splunk instance stop responding

tamduong16 — Tue, 29 Sep 2020 20:39:53 GMT

I tried to change the port to 9089 and now it showing: "This page isn't working" localhost didn't send any data. ERR_EMPTY_RESPONSE

Re: My splunk instance stop responding

hortonew — Thu, 26 Jul 2018 19:58:23 GMT

I would definitely investigate mcafee now - i've seen it prevent a lot of applications on windows from working. If it has a log you can view see if anything is getting denied. If you have the ability to disable it temporarily I'd try that.

Re: My splunk instance stop responding

tamduong16 — Thu, 26 Jul 2018 20:10:20 GMT

Thank you!