Getting Data In

Is Splunk UniversalForwarder 7.0.x on Windows Server 2016 compatible with Docker engine?

ivansha
New Member

I ran into an issue on a Windows Server 2016 which is in company domain with Splunk UF 7.0.7 version installed. When I install Docker engine 17.06.2-ee-16, I can't run any containers on the server. If I uninstall Splunk UF from the server, Docker runs fine and I can create/execute/remove containers just fine.

What I've discovered so far
I prepped the VM per Docker docs (installed Containers feature) and got the engine installed and running. However, as soon as I try to start a container (e.g. hello-world), my Powershell console hangs and never returns control until I either kill docker.exe process or restart docker service.
If I kill docker.exe process and try to list containers (docker ps -a), the Powershell console hangs again. That's the behavior until I restart docker service on the server.
After restarting the service, I can list containers and see that a container I tried to run sometimes ends up with state Created and sometimes Dead but never in running state. If I try to start the container, Powershell console hangs again. If I try to remove the container, I get an error:
Error response from daemon: driver "windowsfilter" failed to remove root filesystem for b3f.........: rename C:\ProgramData\docker\windowsfilter\b3f......... C:\ProgramData\docker\windowsfilter\b3f.........-removing: Access is denied.

The only way to remove the container in limbo state is to reboot Windows server. After that I can remove the container but trying to run gets me into the same cycle.

Since the issue gets resolved after Splunk UF is uninstalled, it appears to me that Splunk has something to do with the fact that Docker engine doesn't function properly. I configured Docker engine to run in debug mode but don't see any errors or warnings in its logs. Has anyone come across such issue? Any hints where I should look on Splunk config or logs side to get a sense of why it may hinder Docker engine?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

If you hear someone at .conf26 shouting "gear down, GEAR DOWN" across the show floor, you have found us.  The ...

Turn Cisco Telemetry Into Action with Cisco Data Fabric, powered by the Splunk ...

The surge in machine data is already hitting enterprise budgets, and the agentic era will only intensify it. ...

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...