Forwarder Ingestion Latency

[Sinfo]

The IP address keeps changing with the same error.

Forwarder Ingestion Latency
Cause(s) d'origine : Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 272246. Message from D97C3DE9-B0CE-408F-9620-5274BAC12C72:192.168.1.191:50409

How do you solve the problem?

[mattymo]

This is part of the splunkd health report.

It is configured in health.conf

Would suggest reviewing if this "forwarder" is sending old files or actually is falling behind or have some clean up needed on its ingestion tracker values.  

[fatsug]

So, how does one isolate the affected forwarder?

The error message reads

Forwarder Ingestion Latency

• Root Cause(s):
  • Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 89. Message from <UUID>:<ip-addrs>:54246
• Unhealthy Instances:

  • indexer1
  • indexer2

  The "message from" section just lists the UUID, an IP adress and a port. Which part here would help me find the actual forwarder? The UUID does not match any "Client name" under forwarder management on the deployment server. The IP adress does not match a server on which I have a forwarder installed.

  One or a few of the indexers are listed as "unhealthy instances" each time. But the actual error sounds like it lives in the forwarder end and not on the indexer.

  With the available information in this warning/error. How can I figure out which forwarder is either experiencing latency issues OR need to have that log file mentioned flushed.