Splunk forwarder does not run any script for a period of time

[cfernaca]

Good morning,

Let me tell you about my case. In my company, we have five indexers, one for development and the other four for production. We have an inputs.conf in a forwarder inside a Docker container python:3.11-slim-bullseye that has three stanzas that execute a script with arguments. One stanza sends the data to development and runs every two minutes, and the other two send the data to production, one running every minute and the other every two minutes.

We have noticed that during a period of time last night, we did not receive any data from the forwarder. Regarding the development stanza, it's correct as the machine was being patched, and Splunk was stopped just during that period. We have observed that during those hours, the forwarder did not execute any scripts. During that time frame, we found these traces in the watchdog.log file of the forwarder:

02-05-2024 20:02:18.220 +0000 ERROR Watchdog - No response received from IMonitoredThread=0x7fabb87fec60 within 8000 ms. Looks like thread name='ExecProcessor' tid=1937852 is busy !? Starting to trace with 8000 ms interval.

Could you please help me understand why the forwarder did not execute any scripts during that time frame?

Thank you very much. Best regards.

[cfernaca]

No, exactly. I have configure two tcpout groups, one for the production servers and the other for development server. The data is not sent to two tcpout groups, only is sent to one. However, the internal agent logs  yes it sent to development and production tcpout groups. It’s possible that the internal agent logs have the behavior you mention?

[isoutamo]

Hi
If I understood correctly, you have one forwarder which are sending those events to different indexers. As those are configured to send one by one based on your input and one target is down it cannot send to that. Based on your outputs.conf, it’s quite probably that it’s just waiting that this target (your dev, which has patched) will be available and then it continue with next.
This is normal issue when you are replicating outputs e.g. to splunk and syslog server.
r. Ismo