Monitoring Splunk

How can I check why splunkd is at 100% CPU?

david_resnick
Explorer

I'm running docker based splunk, version splunk/splunk:7.0.2

At some point I've stopped being able to log in to the UI. After entering my credentials I get the following message:

503 Service Unavailable

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

splunkd is running at 100%+ CPU, which probably explains why I can't access anything after the login.

There are no significant warnings or errors in any logs.

How can I debug this or determine what the problem is here?

Tags (3)
0 Karma

koshyk
Super Champion

can u please put your docker configs here? did you map drives for etc & var to your host?

0 Karma

david_resnick
Explorer
    docker run
      --name splunk
      --hostname splunk
      --detach=true
      -p 80:8000
      -p 8088:8088
      -p 8089:8089
      -p 9998:9998
      -p 9999:9999
      -v /data/var:/opt/splunk/var
      -v /data/var:/opt/splunk/etc
      -e "SPLUNK_START_ARGS=--accept-license --answer-yes"
      {{ splunk_image }}
0 Karma

brdr
Contributor

Assuming your using Linux, you could 'strace' the splunkd process for a short bit and see what it is doing.

strace -p 'PID of splunkd' -o output.txt

Let it run for a few minutes then check output.txt.

0 Karma

deepashri_123
Motivator

Hey@david_resnick,

Can you try restarting splunkd service

0 Karma

david_resnick
Explorer

I did try restarting. I even replace the EC2 instance it's running on (though with the same volume holding var and etc).

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...