Getting Data In
Highlighted

Splunking Windows Server 2016 Docker Containers

Explorer

Hello, I am currently saving my logs into a folder for my Docker containers. From there I installed the Splunk Universal Forwarder on the server and it is ingesting the logs from the path I set. Is this the recommended way of doing this? I saw that I can set a daemon.json file up that will ingest all container logs as well, which sounds like the simpler way of doing it.

This only gives me errors with the containers itself - I would also like to look at the metrics of these containers. I haven't really found any documentation on this - does anyone have any input on the best method to monitor logs and metrics of my Windows Docker Container environment?

Thank you!

0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Splunk Employee
Splunk Employee

Hi zielkepham!

Are you referring to application logs? or windows system logs? both?

Which orchestration engine are you using?

Our early configs of using the UF as a deamonset/universal service can be found here https://github.com/splunk/docker-itmonitoring/blob/7.0.0-k8s/README-k8s.md and I'd be happy to help you get set up and have a look at what is possible and what fits your needs.

We will be meeting with the Windows Container teams soon to look at the best options for our customers, so feel free to reach out to me to be included in any Early Access Programs we might have.

Also come join us on slack (splk.it/slack) in #docker, #kubernetes, or #openshift

Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Explorer

Hi mmodestino!

I want to collect both application logs and windows system logs. I am only collecting Docker application logs currently.

Versions:
Splunk: 6.6.3.2
Docker: 17.03.1-ee-3
Docker-Compose: 1.16.1

I am planning on uninstalling the universal forwarder and using the token method (HEC) of ingesting logs - but still I think that only solves half of the problem I am having. I would like to pull the container metrics as well.

0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Splunk Employee
Splunk Employee

Yeah, I'd say the UF directly on your worker nodes, as a global service, will solve collecting stdout/stderr (ie app logs) while installing in the container image will solve win specific system logs, or app logs that dont send to stderr/stdout

Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Explorer

Mmodestino,

So are you suggesting keeping the Universal Forwarder installed AND use the HEC token method to pull container metrics?

Thanks!

0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Splunk Employee
Splunk Employee

Well, without knowing exactly what/how you plan on collecting the metrics, my short answer is yes.

In my experience thus far, successfully collecting data holistically from container orchestration platforms, will require a multi-phased approach to cover all use cases you will see:

  1. Node level collection - to cover all host data collection, from container logs and metrics, to host logs and metrics and beyond. This can be done by directly installing the UF on the nodes, or by running things like global services and daemonsets, etc.
  2. Deploying a pod(s)/container(s) to the cluster, or installing directly in containers to get logs that dont get spit back to the host. stderr/stdout, etc, or to run api polling/watching etc.

Be glad to talk with you more about specifics, in the slack chat or shoot me an email and we can discuss

Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Explorer

Sure, I would like to discuss this a bit further - where can I find your e-mail?

Thank you!

0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Splunk Employee
Splunk Employee
0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Explorer

Hi Modestino,

I sent you an email last week - wondering if you've seen it?

Thanks!

0 Karma
Highlighted

Re: Splunking Windows Server 2016 Docker Containers

Splunk Employee
Splunk Employee

yep! replied

0 Karma