ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_stats.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[eygtmbot]

Hi All,

We are using Docker Swarm on Ubuntu 16.04
I'm starting my forwarder stack with below YML file.

version: '3'

services:
  splunk-forwarder:
    image: splunk/universalforwarder:7.0.0-monitor
    ports:
      - "514:1514/udp"
    deploy:
      replicas: 2
    environment:
      SPLUNK_START_ARGS: --accept-license --answer-yes
      SPLUNK_FORWARD_SERVER: "server:9997"
      SPLUNK_USER: root
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var

volumes:
  opt-splunk-etc:
  opt-splunk-var:

After running stack deploy

sudo docker stack deploy -c stack-file.yml kiran

All the containers are propagated with the following error, can you please take a look?

04-07-2018 00:52:17.485 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh" /opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh: line 10: docker: command not found
04-07-2018 00:52:17.505 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
04-07-2018 00:52:17.506 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_top.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

What I'm missing? I'm running this on an Ubuntu 16.4 machine.

Is that a permission issue?

I already added ENV SPLUNK_USER: root

I tried sudo usermod -aG docker $USER but that didn't help.

Thanks in advance
Kiran

[outcoldman]

You do not mount /var/run/docker.sock to the container from the host, also few more folders, you can find them in the docs https://hub.docker.com/r/splunk/universalforwarder/

 --volume /var/lib/docker/containers:/host/containers:ro \
 --volume /var/log:/docker/log:ro \
 --volume /var/run/docker.sock:/var/run/docker.sock:ro \

I see that this page also has a docker-compose example, but seems like this example is for general forwarder, not the forwarder with the scripts to collect docker metrics.

---

If you don't mind paid solutions, I would also recommend taking a look at our solution for Monitoring Docker https://www.outcoldsolutions.com. We have a free 30 days trial, instructions on how to install it https://www.outcoldsolutions.com/docs/monitoring-docker/

[eygtmbot]

I didn't try this but I made it work through the Splunk driver instead of syslog. May I know which one is the splunk recommendation. Splunk Forwarder or HTTP listener?

Thanks,
Kiran

[outcoldman]

Splunk Docker Driver is great for a start, but it has some limitations and its issues. As an example, it is not resistant to failures, in case of network failures, it will retry several times, but will give up at some point.
But if you will install Heavy Weight Forwarder as a side container on every host and use it as a target for your logs - you will reduce the possibility of network issues between the logging driver and Splunk. And Splunk itself knows how to buffer data before forwarding.
Using syslog in case if you can use Splunk Logging Driver does not make a lot of sense, because in case of Splunk logging driver you get more Splunk-friendly format, also can add pre-indexed fields from container labels. But there can be situations, where you want to use syslog - if you want to reassemble multiline messages into one.

Saying that - still recommend to look on our solution https://www.outcoldsolutions.com, as we deal with all of these problems, and give you application monitoring on top of that.