Getting Data In

Splunking Windows Server 2016 Docker Containers

zielkepham
Explorer

Hello, I am currently saving my logs into a folder for my Docker containers. From there I installed the Splunk Universal Forwarder on the server and it is ingesting the logs from the path I set. Is this the recommended way of doing this? I saw that I can set a daemon.json file up that will ingest all container logs as well, which sounds like the simpler way of doing it.

This only gives me errors with the containers itself - I would also like to look at the metrics of these containers. I haven't really found any documentation on this - does anyone have any input on the best method to monitor logs and metrics of my Windows Docker Container environment?

Thank you!

0 Karma

outcoldman
Communicator

Hi @zielkepham,

I have not seen yet ready to use solution for Windows.

Our company (https://www.outcoldsolutions.com) is focused on providing monitoring solutions for Docker, OpenShift, Kubernetes environments. We are just starting with Windows support, and have some early prototypes. Feel free to send us an email https://www.outcoldsolutions.com/contact/ so we can talk and understand your needs.

0 Karma

outcoldman
Communicator

Just FYI for other folks, who are looking for solution to monitor Windows Containers - we have finished our solution for Monitoring Windows Containers. We are planning to publish it on SplunkBase and our docs in following days/week.
Please send us email contact@outcoldsolutions.com if you want to get access to beta version.

0 Karma

zielkepham
Explorer

Hi Outcold,

I've looked into your solution for a bit, but noticed it was more for Linux docker containers. I'll send an e-mail your way.

Thanks.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi zielkepham!

Are you referring to application logs? or windows system logs? both?

Which orchestration engine are you using?

Our early configs of using the UF as a deamonset/universal service can be found here https://github.com/splunk/docker-itmonitoring/blob/7.0.0-k8s/README-k8s.md and I'd be happy to help you get set up and have a look at what is possible and what fits your needs.

We will be meeting with the Windows Container teams soon to look at the best options for our customers, so feel free to reach out to me to be included in any Early Access Programs we might have.

Also come join us on slack (splk.it/slack) in #docker, #kubernetes, or #openshift

- MattyMo

zielkepham
Explorer

Hi mmodestino!

I want to collect both application logs and windows system logs. I am only collecting Docker application logs currently.

Versions:
Splunk: 6.6.3.2
Docker: 17.03.1-ee-3
Docker-Compose: 1.16.1

I am planning on uninstalling the universal forwarder and using the token method (HEC) of ingesting logs - but still I think that only solves half of the problem I am having. I would like to pull the container metrics as well.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Yeah, I'd say the UF directly on your worker nodes, as a global service, will solve collecting stdout/stderr (ie app logs) while installing in the container image will solve win specific system logs, or app logs that dont send to stderr/stdout

- MattyMo

zielkepham
Explorer

Mmodestino,

So are you suggesting keeping the Universal Forwarder installed AND use the HEC token method to pull container metrics?

Thanks!

0 Karma

mattymo
Splunk Employee
Splunk Employee

Well, without knowing exactly what/how you plan on collecting the metrics, my short answer is yes.

In my experience thus far, successfully collecting data holistically from container orchestration platforms, will require a multi-phased approach to cover all use cases you will see:

  1. Node level collection - to cover all host data collection, from container logs and metrics, to host logs and metrics and beyond. This can be done by directly installing the UF on the nodes, or by running things like global services and daemonsets, etc.
  2. Deploying a pod(s)/container(s) to the cluster, or installing directly in containers to get logs that dont get spit back to the host. stderr/stdout, etc, or to run api polling/watching etc.

Be glad to talk with you more about specifics, in the slack chat or shoot me an email and we can discuss

- MattyMo

zielkepham
Explorer

Sure, I would like to discuss this a bit further - where can I find your e-mail?

Thank you!

0 Karma

mattymo
Splunk Employee
Splunk Employee
0 Karma

zielkepham
Explorer

Hi Modestino,

I sent you an email last week - wondering if you've seen it?

Thanks!

0 Karma

mattymo
Splunk Employee
Splunk Employee

yep! replied

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...