Why do the indexing time of all UF logs gets delayed, including the internal logs (This delay occurs every 30mins)?

[splunkis0927]

I have a issue:

On one of my UF,  The indexing time of all the logs (including the internal logs) get delayed for 2-3mins, and This delay occurs every 30mins. other UFs looks ok.

• we have checked the queue on this UF is not blocked.
• we have changed [thruput]maxKBps = 0

But the indexing time issue is still there. Can anyone please help with this issue ?

Do we need to check more configs or logs?

When indexing time get delayed I can see logs below: 

[logs]:

• INFO Watchdog - No response received from IMonitoredThread=0xxxxxxxxx within elapsed=8000 ms. Looks like thread_name="TcpOutEloop" thread_id=1xxxx is busy !? Starting to trace with timeout=8000 ms interval.

• INFO Watchdog - Stopping trace. Response for IMonitoredThread ptr=0xxxxxxxxx - thread_name="TcpOutEloop" thread_id=1xxxx - finally received after 3xxxx ms (estimation only).

• INFO HealthChangeReporter - feature="Ingestion Latency" indicator="ingestion_latency_lag_sec" previous_color=green color=yellow due_to_threshold_value=15 measured_value=30 reason="Events from tracker.log are delayed for 30 seconds, which is more than the yellow threshold (15 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked."

Possible is the UF being used to monitor too many files at the same time?  so , make 

thread name='TcpOutEloop' busy ?

[scelikok]

Hi @splunkis0927,

According to the below log, your UF cannot reach at least one of your indexers. 

• WARN AutoLoadBalancedConnectionStrategy [1xxxx TcpOutEloop] - Cooked connection to ip=xxx.xx.xx.xx:9997 timed out

30 seconds is the default load balance timer for UF. Whenever this UF tries connecting particular indexers or indexers, it waits for the timeout. During this period UF may not be able to send logs, after connecting to another indexer you see the logs but delayed.

Please check the connectivity, routing, firewall, etc between this UF and indexers.

[splunkis0927]

Thanks for the reply.

After double checking UFs,  the WARN below seems not there every time when indexing time get delayed.

I have edit my post. 

WARN AutoLoadBalancedConnectionStrategy [1xxxx TcpOutEloop] - Cooked connection to ip=xxx.xx.xx.xx:9997 timed out
 

Possible is the UF being used to monitor too many files at the same time?  so , make 

thread name='TcpOutEloop' busy ?