Getting Data In

Why do the indexing time of all UF logs gets delayed, including the internal logs (This delay occurs every 30mins)?

splunkis0927
Engager

I have a issue:

On one of my UF,  The indexing time of all the logs (including the internal logs) get delayed for 2-3mins, and This delay occurs every 30mins. other UFs looks ok.

  • we have checked the queue on this UF is not blocked.
  • we have changed [thruput]maxKBps = 0

 

But the indexing time issue is still there. Can anyone please help with this issue ?

Do we need to check more configs or logs?

 

When indexing time get delayed I can see logs below: 

[logs]:

  • INFO Watchdog - No response received from IMonitoredThread=0xxxxxxxxx within elapsed=8000 ms. Looks like thread_name="TcpOutEloop" thread_id=1xxxx is busy !? Starting to trace with timeout=8000 ms interval.

 

  • INFO Watchdog - Stopping trace. Response for IMonitoredThread ptr=0xxxxxxxxx - thread_name="TcpOutEloop" thread_id=1xxxx - finally received after 3xxxx ms (estimation only).

 

  • INFO HealthChangeReporter - feature="Ingestion Latency" indicator="ingestion_latency_lag_sec" previous_color=green color=yellow due_to_threshold_value=15 measured_value=30 reason="Events from tracker.log are delayed for 30 seconds, which is more than the yellow threshold (15 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked."

 

Possible is the UF being used to monitor too many files at the same time?  so , make 

thread name='TcpOutEloop' busy ?

 

Labels (2)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @splunkis0927,

According to the below log, your UF cannot reach at least one of your indexers. 

  • WARN AutoLoadBalancedConnectionStrategy [1xxxx TcpOutEloop] - Cooked connection to ip=xxx.xx.xx.xx:9997 timed out

30 seconds is the default load balance timer for UF. Whenever this UF tries connecting particular indexers or indexers, it waits for the timeout. During this period UF may not be able to send logs, after connecting to another indexer you see the logs but delayed.

Please check the connectivity, routing, firewall, etc between this UF and indexers.

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

splunkis0927
Engager

Thanks for the reply.

After double checking UFs,  the WARN below seems not there every time when indexing time get delayed.

I have edit my post. 

WARN AutoLoadBalancedConnectionStrategy [1xxxx TcpOutEloop] - Cooked connection to ip=xxx.xx.xx.xx:9997 timed out
 

Possible is the UF being used to monitor too many files at the same time?  so , make 

thread name='TcpOutEloop' busy ?

0 Karma
Get Updates on the Splunk Community!

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...