My splunk instance stop responding

[tamduong16]

I have been running splunk on a server for sometime now and last week it stops responding. I don't know what could be the issue that cause it. When I access the server and go to localhost:8000, the page keeps loading and never give me any message. What can I do now?

[tamduong16]

I tried to change the port to 9089 and now it showing: "This page isn't working" localhost didn't send any data. ERR_EMPTY_RESPONSE

[hortonew]

I would definitely investigate mcafee now - i've seen it prevent a lot of applications on windows from working. If it has a log you can view see if anything is getting denied. If you have the ability to disable it temporarily I'd try that.

[tamduong16]

Thank you!

[woodcock]

You probably ran out of disk space. What does df show you?

[tamduong16]

I checked disk space and I still have at least 10 GB of disk space left.

[tamduong16]

I tried to change the port to 9089 and now it showing: "This page isn't working" ERR_EMPTY_RESPONSE

[hortonew]

Has it created a socket on port 8000 yet? Try: netstat -an | select-string 8000 and see if anything returns that it's listening. If not, do you have anything denying it from creating that socket? Anti-virus, etc?

[tamduong16]

I tried that command but it said select-string is not recognized as an internal or external command. I have McAfee running but I don't know if that blocking the port. How can I find out?

[hortonew]

Run command via powershell, not cmd.exe if that's what you were doing. select-string is a powershell command.

[akocak]

in CMD:
netstat -an | findstr /s /i 8000

[tamduong16]

I did that and it display a list of ip addresss along with a column to the right that indicate FIN_WAIT_2 and CLOSE_WAIT

[hortonew]

But nothing with LISTENING at the end? You should see an entry like:

tcp 0 0.0.0.0:8000 0.0.0.0:* LISTEN

So: netstat -an | select-string 8000 | select-string LIST

All the entries about fin/close are IPs trying to hit that socket and those connections being torn down.

[pradeepkumarg]

Did you try restarting?
Any errors in splunkd.log? Is the server healthy? Try using servername:8000 or serverip:8000 and see if it makes any difference?

[tamduong16]

I did try to stopped, restarted it but even when it stop, my localhost:8000 still doesn't response. I would expect it to send me something like server's not responding. The server is healthy. There are 2 errors such as: ERROR HttpListener - Exception. I just tried using servername:8000 and serverip:8000 and it still doesn't work.

[akocak]

Can you give more information about your OS and version of the Splunk ?

[tamduong16]

Hi I'm running version 6.6.2. And it is running on a windows server.