Hello, I'm trying to install splunk logging driver on my swarm master. Here is the stack file I'm using version: '3' services: web: image: "${DOCKER_IMAGE_PATH}/:${RELEASE_TAG}" ports: - "8080:8080" - "8081:80" deploy: replicas: 3 build: context: ../../ dockerfile: ../Dockerfile-js environment: - PORT=80 # restart: unless-stopped logging: driver: splunk options: splunk-url: "${SPLUNK_URL}" splunk-token: "${SPLUNK_TOKEN}" splunk-capath: "/mnt/splunk-ca-cert.pem" splunk-caname: "${SPLUNK_URL}" splunk-insecureskipverify: "false" tag: "{{.Name}}/{{.ID}}" labels: "NEurope" env: "${TARGET_NAME}" When I'm running docker stack create -c filname.txt stack_name its searching for splunk-capath: "/mnt:/mnt/splunk-ca-cert.pem file in local system. I already copied this file in the container(when image built) same location but stack file is searching in the system instead of container. How can i make my stack file to look for the file inside the container??? Any help will be appreciated !!! Thanks ... View more

Hello, We implemented collecting Docker logs using splunk logging driver, It pushes the docker logs very well and good. But we have a bigger problem now. Let's consider my Splunk-Indexor is down while spinning up docker containers, those containers will not be able to establish the connection with Splunk-indexor machine. Now that's going to crash entire docker engine on the system and you will not be able to execute any of the docker commands in those machines, this will hang up the entire docker engine in the machine. To fix this I had to restart the VM, docker service restart is not helping. How can I mitigate this error? Is this the docker issue or the Splunk one? Here is the swarm-stack file I'm using version: '3' services: worker: image: "${DOCKER_IMAGE_PATH}/worker:${RELEASE_TAG}" deploy: replicas: 3 build: context: ../../ dockerfile: ../Dockerfile-worker environment: ports: - "8083:3000" logging: driver: splunk options: splunk-url: "${SPLUNK_URL}" splunk-token: "${SPLUNK_TOKEN}" splunk-insecureskipverify: "true" tag: "{{.Name}}/{{.ID}}" labels: "NEurope" env: "${TARGET_NAME}" If the Splunk driver works like this, then I need to rebuild/restart Docker Containers each and every time if there is a restart on the Splunk server(Indexor) Thanks, Kiran ... View more

Hi All, We are using Docker Swarm on Ubuntu 16.04 I'm starting my forwarder stack with below YML file. version: '3' services: splunk-forwarder: image: splunk/universalforwarder:7.0.0-monitor ports: - "514:1514/udp" deploy: replicas: 2 environment: SPLUNK_START_ARGS: --accept-license --answer-yes SPLUNK_FORWARD_SERVER: "server:9997" SPLUNK_USER: root volumes: - opt-splunk-etc:/opt/splunk/etc - opt-splunk-var:/opt/splunk/var volumes: opt-splunk-etc: opt-splunk-var: After running stack deploy sudo docker stack deploy -c stack-file.yml kiran All the containers are propagated with the following error, can you please take a look? 04-07-2018 00:52:17.485 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh" /opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh: line 10: docker: command not found 04-07-2018 00:52:17.505 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 04-07-2018 00:52:17.506 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_top.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? What I'm missing? I'm running this on an Ubuntu 16.4 machine. Is that a permission issue? I already added ENV SPLUNK_USER: root I tried sudo usermod -aG docker $USER but that didn't help. Thanks in advance Kiran ... View more

https://www.splunk.com/blog/2015/08/24/collecting-docker-logs-and-stats-with-splunk.html With reference to this documentation, I'm trying to forward my container logs to Splunk forwarder listening on 514 port. Forwarder Config version: '2' volumes: opt-splunk-etc: opt-splunk-var: services: splunkuniversalforwarder: hostname: splunkuniversalforwarder image: splunk/universalforwarder:7.0.0 environment: SPLUNK_START_ARGS: --accept-license --answer-yes SPLUNK_FORWARD_SERVER: "hostname:9997" volumes: - opt-splunk-etc:/opt/splunk/etc - opt-splunk-var:/opt/splunk/var ports: - "514:1514/udp" - "8000:8000" - "9997:9997" - "8088:8088" after starting the container I'm running docker exec -it splunk_forwarder_1 entrypoint.sh splunk add udp 1514 -sourcetype syslog but its giving a faliur message says root@splunk-forwarder:/home/splunk/docker-forwarer# docker exec -it dockerforwarer_splunkuniversalforwarder_1 entrypoint.sh splunk add udp 1514 -sourcetype syslog Splunk username: admin Password: Failed to create. Configuration for port 1514 already exists. Splunk forwarder listerning on 514 syslog port root@splunk-forwarder:/home/splunk/docker-forwarer# netstat -lnp | grep 514 udp6 0 0 :::514 :::* 87192/docker-proxy Here is the NGINX server I'm trying to forward logs from nginx: image: nginx ports: - 80:80 - 443:443 volumes_from: - vdata restart: always log_driver: syslog log_opt: syslog-tag: nginxproxy_nginx syslog-address: udp://127.0.0.1:514 When I'm starting the NGINX container its stuk on conneting to the syslog-address: udp://127.0.0.1:514 can you please let us know what I'm doing worng ? Thanks, ... View more