Splunk web interface goes down after couple of minutes

[fedayn05]

Hello Splunkers,

I am new to splunk , I am using Splunk Entreprise 10.0 running on ubuntu,  and I seem to be having an issue where the web interface stops after a few minutes(sometimes up to 3 hours). I then must issue "splunk start" to bring it back but it stops working again after awhile.The issue is that It is not indexing during that time off. 

Do you please have any idea or suggestions about this issue?

Thank you so much for you attention.

[isoutamo]

Is the splunkd still up and running or is it also crashed?

[livehybrid]

Hi @fedayn05 

What is the spec of the server you are running this on? mainly the CPU Cores/RAM/Disk IOPs?

The reference hardware (https://help.splunk.com/en/splunk-enterprise/get-started/deployment-capacity-manual/9.4/performance-...) calls for 12 cores and 12 GB RAM, 800 IOPS

As others have said, the problem here sounds like a resource constraint which is killing the process. I presume it warns about cleaning up a PID file when it starts too?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[fedayn05]

Hello All,

Thank you for your answers, i guess the issue is not related to resources , as the following is the state of my server : 

- CPU : 8% used, 88% idle 

- RAM : 2 Go used out of 31 Go

- Disk : 13% used

[kiran_panchavat]

@fedayn05 

Check the _internal index for the logs in web_service.log. Do you see anything prior to the stopping ? check the "Out Of Memory / OOM" events, the system can kill a process.

grep -i 'oom' /var/log/syslog 
dmesg | grep -i 'oom'

dmesg -T | egrep -i 'killed process'

[fedayn05]

Hello Kiran,

Thank you for your answer, I run the commands you provided , and the issue is not related to "Out Of Memory / OOM" events or the system killing the process

[kiran_panchavat]

@fedayn05 Can you pls check the web_service.log

[fedayn05]

Hello Kiran,

I found an error that i did not understand actually : 

ERROR [68e3932b467f7df432adc0] startup:116 - Unable to read in product version information; isSessionKeyDefined=True error=[HTTP 401] Client is not authentic.

But i did not understand it.

[PickleRick]

If "splunk start" "resolves" the issue, that means that whole splunkd process crashes. It can be caused by many different things and requires more thorough investigation. It can be anything from out-of-memory killer to faulty hardware.