Hello,
We implemented collecting Docker logs using splunk logging driver, It pushes the docker logs very well and good. But we have a bigger problem now.
Let's consider my Splunk-Indexor is down while spinning up docker containers, those containers will not be able to establish the connection with Splunk-indexor machine. Now that's going to crash entire docker engine on the system and you will not be able to execute any of the docker commands in those machines, this will hang up the entire docker engine in the machine. To fix this I had to restart the VM, docker service restart is not helping.
How can I mitigate this error?
Is this the docker issue or the Splunk one?
Here is the swarm-stack file I'm using
version: '3'
services:
worker:
image: "${DOCKER_IMAGE_PATH}/worker:${RELEASE_TAG}"
deploy:
replicas: 3
build:
context: ../../
dockerfile: ../Dockerfile-worker
environment:
ports:
- "8083:3000"
logging:
driver: splunk
options:
splunk-url: "${SPLUNK_URL}"
splunk-token: "${SPLUNK_TOKEN}"
splunk-insecureskipverify: "true"
tag: "{{.Name}}/{{.ID}}"
labels: "NEurope"
env: "${TARGET_NAME}"
If the Splunk driver works like this, then I need to rebuild/restart Docker Containers each and every time if there is a restart on the Splunk server(Indexor)
Thanks,
Kiran
... View more