Monitoring Splunk
Highlighted

How can I check why splunkd is at 100% CPU?

Explorer

I'm running docker based splunk, version splunk/splunk:7.0.2

At some point I've stopped being able to log in to the UI. After entering my credentials I get the following message:

503 Service Unavailable

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

splunkd is running at 100%+ CPU, which probably explains why I can't access anything after the login.

There are no significant warnings or errors in any logs.

How can I debug this or determine what the problem is here?

Tags (3)
0 Karma
Highlighted

Re: How can I check why splunkd is at 100% CPU?

Motivator

Hey@david_resnick,

Can you try restarting splunkd service

0 Karma
Highlighted

Re: How can I check why splunkd is at 100% CPU?

Explorer

I did try restarting. I even replace the EC2 instance it's running on (though with the same volume holding var and etc).

0 Karma
Highlighted

Re: How can I check why splunkd is at 100% CPU?

Super Champion

can u please put your docker configs here? did you map drives for etc & var to your host?

0 Karma
Highlighted

Re: How can I check why splunkd is at 100% CPU?

Explorer
    docker run
      --name splunk
      --hostname splunk
      --detach=true
      -p 80:8000
      -p 8088:8088
      -p 8089:8089
      -p 9998:9998
      -p 9999:9999
      -v /data/var:/opt/splunk/var
      -v /data/var:/opt/splunk/etc
      -e "SPLUNK_START_ARGS=--accept-license --answer-yes"
      {{ splunk_image }}
0 Karma
Highlighted

Re: How can I check why splunkd is at 100% CPU?

Contributor

Assuming your using Linux, you could 'strace' the splunkd process for a short bit and see what it is doing.

strace -p 'PID of splunkd' -o output.txt

Let it run for a few minutes then check output.txt.

0 Karma