All Apps and Add-ons

How to send Docker containers logs to Splunk?

Ranjithatnutani
New Member

We're running two processes Nginx and an application inside Docker container. Both the process lifecycles are managed through the Supervisor. I want to understand if I use Docker Log driver for Splunk, will Splunk be able to automatically forward both the process logs indexer.

Tags (1)
0 Karma

outcoldman
Communicator

It is a hard question, as it depends on how you setup the Supervisor.

Let me start from the beginning. Running multiple processes in the same container is an anti-pattern. Try to avoid it as much as possible. Kubernetes, for example, have a great solution for your case, where they can deploy two containers in the same Pod and just setup communication between the containers on the same loopback network interface (127.0.0.1), so for the processes, it will look like they are running in the same container. See https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-v... for details.

Unfortunately, Docker does not have this out of the box, and configuring the same can be problematic.

There are two options. You can redirect stdout and stderr from the processes to the supervisord stdout/stderr (took an example from https://github.com/CachetHQ/Docker/blob/master/conf/supervisord.conf#L18-L25)

[program:nginx]
command=/usr/sbin/nginx
stdout_events_enabled=true
stderr_events_enabled=true
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
stdout_logfile=/dev/stdout
stderr_logfile=/dev/stderr

In that case, all the logs will end up in container stdout/stderr. You can use Docker Logger Drivers to send these logs to Splunk. The problem with this approach - all the logs, from supervisord, from nginx and all other processes, will end up in container stdout/stderr, it could be hard to distinguish them.

Another approach is to have a data volume for the application logs. Attach the same volume to the Splunk Universal Forwarder or another process (like our collector https://www.outcoldsolutions.com) and forward these logs as you usually do. Don't forget to set the log rotation.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...