Monitoring Splunk

How can I check why splunkd is at 100% CPU?


I'm running docker based splunk, version splunk/splunk:7.0.2

At some point I've stopped being able to log in to the UI. After entering my credentials I get the following message:

503 Service Unavailable

The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running.

splunkd is running at 100%+ CPU, which probably explains why I can't access anything after the login.

There are no significant warnings or errors in any logs.

How can I debug this or determine what the problem is here?

Tags (3)
0 Karma

Super Champion

can u please put your docker configs here? did you map drives for etc & var to your host?

0 Karma

    docker run
      --name splunk
      --hostname splunk
      -p 80:8000
      -p 8088:8088
      -p 8089:8089
      -p 9998:9998
      -p 9999:9999
      -v /data/var:/opt/splunk/var
      -v /data/var:/opt/splunk/etc
      -e "SPLUNK_START_ARGS=--accept-license --answer-yes"
      {{ splunk_image }}
0 Karma


Assuming your using Linux, you could 'strace' the splunkd process for a short bit and see what it is doing.

strace -p 'PID of splunkd' -o output.txt

Let it run for a few minutes then check output.txt.

0 Karma



Can you try restarting splunkd service

0 Karma


I did try restarting. I even replace the EC2 instance it's running on (though with the same volume holding var and etc).

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...