Deployment Architecture

My splunk instance stop responding

tamduong16
Contributor

I have been running splunk on a server for sometime now and last week it stops responding. I don't know what could be the issue that cause it. When I access the server and go to localhost:8000, the page keeps loading and never give me any message. What can I do now?

0 Karma

tamduong16
Contributor

I tried to change the port to 9089 and now it showing: "This page isn't working" localhost didn't send any data. ERR_EMPTY_RESPONSE

0 Karma

hortonew
Builder

I would definitely investigate mcafee now - i've seen it prevent a lot of applications on windows from working. If it has a log you can view see if anything is getting denied. If you have the ability to disable it temporarily I'd try that.

0 Karma

tamduong16
Contributor

Thank you!

0 Karma

woodcock
Esteemed Legend

You probably ran out of disk space. What does df show you?

0 Karma

tamduong16
Contributor

I checked disk space and I still have at least 10 GB of disk space left.

0 Karma

tamduong16
Contributor

I tried to change the port to 9089 and now it showing: "This page isn't working" ERR_EMPTY_RESPONSE

0 Karma

hortonew
Builder

Has it created a socket on port 8000 yet? Try: netstat -an | select-string 8000 and see if anything returns that it's listening. If not, do you have anything denying it from creating that socket? Anti-virus, etc?

0 Karma

tamduong16
Contributor

I tried that command but it said select-string is not recognized as an internal or external command. I have McAfee running but I don't know if that blocking the port. How can I find out?

0 Karma

hortonew
Builder

Run command via powershell, not cmd.exe if that's what you were doing. select-string is a powershell command.

0 Karma

akocak
Contributor

in CMD:
netstat -an | findstr /s /i 8000

tamduong16
Contributor

I did that and it display a list of ip addresss along with a column to the right that indicate FIN_WAIT_2 and CLOSE_WAIT

0 Karma

hortonew
Builder

But nothing with LISTENING at the end? You should see an entry like:

tcp 0 0.0.0.0:8000 0.0.0.0:* LISTEN

So: netstat -an | select-string 8000 | select-string LIST

All the entries about fin/close are IPs trying to hit that socket and those connections being torn down.

0 Karma

pradeepkumarg
Influencer

Did you try restarting?
Any errors in splunkd.log? Is the server healthy? Try using servername:8000 or serverip:8000 and see if it makes any difference?

0 Karma

tamduong16
Contributor

I did try to stopped, restarted it but even when it stop, my localhost:8000 still doesn't response. I would expect it to send me something like server's not responding. The server is healthy. There are 2 errors such as: ERROR HttpListener - Exception. I just tried using servername:8000 and serverip:8000 and it still doesn't work.

0 Karma

akocak
Contributor

Can you give more information about your OS and version of the Splunk ?

0 Karma

tamduong16
Contributor

Hi I'm running version 6.6.2. And it is running on a windows server.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...