Why is Splunk forwarder preventing Docker rebuild?

[nathanluke86]

I am wondering if anyone has come accross this issue before:

System and application versions:
• Docker version 18.09.4
• Splunk version 7.2.6 (?)
• Windows Server 2019 1809 Build

A summary of what we’ve discovered and background information:
• Splunk seems to prevent docker from starting docker containers, they are stuck in a “Created” state
• We do not use Splunk explicitly as our docker logging service, i.e. Splunk is not referenced in any docker config
• Docker and the SplunkForwarder service both start up on host boot
• Changing the dependencies on the service (i.e. docker start first or splunk start first) doesn’t fix the issue
• Stopping splunk while docker is running and then creating the containers works
o As soon as one container has started successfully, we can start splunk and still create more containers
• Restarting splunk while docker is running and then creating the containers does not work

Steps to reproduce on a machine:
1. Boot server up, docker and splunk start automatically
2. Attempt to run docker-compose to create our containers with no containers already running or in an exited state, docker gets stuck with containers in a “Created” state

Steps to mitigate issue:
1. When there are no containers running, stop the splunk service
2. Run docker-compose to create at least one container successfully
3. Start the splunk service
4. Run docker-compose to bring up any remaining containers

Any help or ideas to get a work around would be appreciated
TIA

[uiliammello]

Hi guys!

Someone finds any workaround for this problem?

I have same symptoms here, and I pretend to open a support ticket to Splunk (if anyone had a workaround), but I'm not sure how to collect these identified evidences that you  pointed in this forum (identification of Splunk locking \device\namedpipe\). Can you help me to run this debugging, to generate these troubleshooting logs and send it to Splunk support  ?

Regards,

Uiliam Mello

[Amksa86]

Hello Nathan, 

Did you guys figure out a solution for that? we're having same exact issue, we thing that splunk is locking \device\namedpipe\ and docker is also using that and we used the debugger found that there's an access denied happening when splunk runs on system. 

We also stopped splunk on the system and we get docker to run. 

we're running Splunk UF 8.0.4. 

---Docker  Version : 

Client: Mirantis Container Runtime
Version: 20.10.6
API version: 1.41
Go version: go1.13.15
Git commit: b3766ff
Built: 06/29/2021 17:14:16
OS/Arch: windows/amd64
Context: default
Experimental: true

If you guys have something to help us with please let us know? 

[kentsaunders]

Amksa86, I am experiencing the same problem.  Did you find a solution?

Splunk Universal Forwarder 8.2.6

Windows Core 2019 1809

Client: Mirantis Container Runtime
Version: 20.10.9
API version: 1.41
Go version: go1.16.12m2
Git commit: 591094d
Built: 12/21/2021 21:34:30
OS/Arch: windows/amd64
Context: default
Experimental: true

[Amksa86]

we excluded windows application logs from the inputs.conf file and we get to at least monitor system and security logs. after we added the exclusion the issue stopped and we're able to run both.