I am creating a Splunk forwarder docker container to forward the logs to splunk on coreos. I am able to create a container but the logs are not able to forward to the splunk. I see the below error in splunkd.log.

[vj5]

05-01-2018 21:56:45.851 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_stats.sh" See '/opt/splunk/etc/apps/ta-dockerstats/bin/docker stats --help'.
05-01-2018 21:56:45.851 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_stats.sh" Usage: docker stats [OPTIONS] CONTAINER [CONTAINER...]
05-01-2018 21:56:45.851 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_stats.sh" Display a live stream of container(s) resource usage statistics
05-01-2018 21:56:45.872 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon. Is the docker daemon running on this host?
05-01-2018 21:56:46.810 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon. Is the docker daemon running on this host?
05-01-2018 21:56:47.813 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon. Is the docker daemon running on this host?
05-01-2018 21:56:48.816 +0000 ERROR ExecProcessor - message f

[outcoldman]

Curious, have you seen our solutions for monitoring Docker, Kubernetes and OpenShift clusters? https://www.outcoldsolutions.com/
We also have a blog post explaining how to set up our solution on Tectonic https://www.outcoldsolutions.com/blog/2018-03-21-monitoring-tectonic-in-splunk/

[xpac]

The error you're seeing is from the ta-dockerstats addon you can find here on GitHub.

This add-on is most likely meant to be run on a docker host, not inside a container. It's supposed to collect statistics about running docker containers etc, so I wonder why this is running inside your container?

Did you built your Splunk UF container yourself, or are you using a premade one?

[vj5]

Developers are creating a symlinks to for the application logs in the pods. I want to forward those logs to Splunk using splunk universal forwarder. Here is my inputs.conf. But I don't see any logs forwarded to the splunk UI.
Any help is appreciated.

[monitor:///d/s/r/*.log]
host = hostname
disabled = false
index = indexname
sourcetype = splunk
followSymlink = true

[xpac]

Did you try to access those logs as the user Splunk runs at, to make sure it's not a permission issue?
If that is fine, try /opt/splunkforwarder/bin/splunk list inputstatus to see the status of all of your inputs - you should see your monitor there and also it's status.

[vj5]

@xpac Thanks for your time. I am getting the below output when I am trying /opt/splunkforwarder/bin/splunk list inputstatus this command. Any help is appreciated.

            /docker/log/containers/d.log
    parent = /docker/log/containers/*.log
    type = broken symlink

[xpac]

Yeah, the broken symlink says that your... symlink is broken 😄
You should check with your docker admin who set up that link from the outside into the containers, because it obviously doesn't work. I've too little knowledge on docker to fix that, but if you login as the user Splunk is running as, and do a less /docker/log/containers/d.log, you should get an error message, too. Therefore, the file is simply not accessible, which is an OS/filesystem issue, not a Splunk issue.

[vj5]

when I do less /docker/log/containers/d.log I see output as no such file or directory as output. I see logs are not persistent they are removed or moved every minute or so.

[vj5]

Yes, I am able to access those logs using splunk user. Its now a permission issue.