Hello, I am currently saving my logs into a folder for my Docker containers. From there I installed the Splunk Universal Forwarder on the server and it is ingesting the logs from the path I set. Is this the recommended way of doing this? I saw that I can set a daemon.json file up that will ingest all container logs as well, which sounds like the simpler way of doing it.
This only gives me errors with the containers itself - I would also like to look at the metrics of these containers. I haven't really found any documentation on this - does anyone have any input on the best method to monitor logs and metrics of my Windows Docker Container environment?
Are you referring to application logs? or windows system logs? both?
Which orchestration engine are you using?
Our early configs of using the UF as a deamonset/universal service can be found here https://github.com/splunk/docker-itmonitoring/blob/7.0.0-k8s/README-k8s.md and I'd be happy to help you get set up and have a look at what is possible and what fits your needs.
We will be meeting with the Windows Container teams soon to look at the best options for our customers, so feel free to reach out to me to be included in any Early Access Programs we might have.
Also come join us on slack (splk.it/slack) in #docker, #kubernetes, or #openshift
I want to collect both application logs and windows system logs. I am only collecting Docker application logs currently.
I am planning on uninstalling the universal forwarder and using the token method (HEC) of ingesting logs - but still I think that only solves half of the problem I am having. I would like to pull the container metrics as well.
Yeah, I'd say the UF directly on your worker nodes, as a global service, will solve collecting stdout/stderr (ie app logs) while installing in the container image will solve win specific system logs, or app logs that dont send to stderr/stdout
So are you suggesting keeping the Universal Forwarder installed AND use the HEC token method to pull container metrics?
Well, without knowing exactly what/how you plan on collecting the metrics, my short answer is yes.
In my experience thus far, successfully collecting data holistically from container orchestration platforms, will require a multi-phased approach to cover all use cases you will see:
Be glad to talk with you more about specifics, in the slack chat or shoot me an email and we can discuss