How to monitor a application log path present inside a container in kubernetes?

[deepak_babu]

Currently I'm using Splunk connect for kubernetes to index console logs. Is there a way to index logs that is present inside without without using a sidecar container?
If sidecar is the only option, is there any documentation on how to implement it?

[mattymo]

Hi!

One option is to define an emptyDir( https://kubernetes.io/docs/concepts/storage/volumes/#emptydir ) for your pod to log to, which you then mount into the logging pod. This allows you to simply update the configMap to add a new file input to fluentd. emptydir logs are usually at /var/lib/kubeleton the node, I believe, so you would add a new tail_in and filters as necessary.

One thing I will say...if you logs dont come out of the container runtime (stdout/stderr) and are very high volume (ie, 4000K+ eps) then I would look at the UF docker image instead, and deploy once UF per node for high volume non docker/containerd logging. While Connect for K8s is better at shaping the data for Splunk, the UF whips it in perf. So depends on what the constraints/requirements are.

Will try and post a blog on this and update here....

[DukeScottWu]

@mattymo  we just have the question related to non-container logging issue.  We planned to use UF as sidecar in our service following https://github.com/splunk/docker-splunk/blob/develop/docs/SETUP.md  but want to know which URL and UID/pwd shall be used. The document just gives a blur information.

[mattymo]

Hi, the UF does not have a UI. What exact image are you trying to use? Splunk/splunk or splunk/universalforwarder?

https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md#runtime-configuration

[DukeScottWu]

Hey @mattymo 

we have deployed Splunk Enterprise 8 and now want to deploy the UF as a sidecar with our service container to ship the logs to splunk server. so my original question is how to configure the UF in docker? and is there any special configuration required in the splunk enterprise side?

btw: appreciated for your quick response 🙂

[mattymo]

No special config needed on splunk side..configuration of the docker image is all done via environment variables or default.yml that passes to ansible. Also you can load splunk apps in as well to handle splunk specific configs. See the advanced document I linked.