How can I check why splunkd is at 100% CPU?

[david_resnick]

I'm running docker based splunk, version splunk/splunk:7.0.2

At some point I've stopped being able to log in to the UI. After entering my credentials I get the following message:

503 Service Unavailable

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

splunkd is running at 100%+ CPU, which probably explains why I can't access anything after the login.

There are no significant warnings or errors in any logs.

How can I debug this or determine what the problem is here?

[koshyk]

can u please put your docker configs here? did you map drives for etc & var to your host?

[david_resnick]

    docker run
      --name splunk
      --hostname splunk
      --detach=true
      -p 80:8000
      -p 8088:8088
      -p 8089:8089
      -p 9998:9998
      -p 9999:9999
      -v /data/var:/opt/splunk/var
      -v /data/var:/opt/splunk/etc
      -e "SPLUNK_START_ARGS=--accept-license --answer-yes"
      {{ splunk_image }}

[brdr]

Assuming your using Linux, you could 'strace' the splunkd process for a short bit and see what it is doing.

strace -p 'PID of splunkd' -o output.txt

Let it run for a few minutes then check output.txt.

[deepashri_123]

Hey@david_resnick,

Can you try restarting splunkd service

[david_resnick]

I did try restarting. I even replace the EC2 instance it's running on (though with the same volume holding var and etc).