Why are we receiving this ingestion latency error after updating to 8.2.1?

[Marc_Williams]

So we just updated to 8.2.1 and we are now getting an Ingestion Latency error…

How do we correct it? Here is what the link says and then we have an option to view the last 50 messages...

 Ingestion Latency

• Root Cause(s):
  • Events from tracker.log have not been seen for the last 6529 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.
  • Events from tracker.log are delayed for 9658 seconds, which is more than the red threshold (180 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.
• Generate Diag?If filing a support case, click here to generate a diag.

Here are some examples of what is shown as the messages:

• 07-01-2021 09:28:52.276 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\var\spool\splunk.
• 07-01-2021 09:28:52.276 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\var\run\splunk\search_telemetry.
• 07-01-2021 09:28:52.276 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\var\log\watchdog.
• 07-01-2021 09:28:52.276 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\var\log\splunk.
• 07-01-2021 09:28:52.276 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\var\log\introspection.
• 07-01-2021 09:28:52.275 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\Splunk\etc\splunk.version.

07-01-2021 09:28:52.269 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\CrushFTP9\CrushFTP.log.

• 07-01-2021 09:28:52.268 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\watchdog\watchdog.log*.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\splunk_instrumentation_cloud.log*.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\license_usage_summary.log.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\introspection.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
• 07-01-2021 09:28:52.267 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\tracker.log*.
• 07-01-2021 09:28:52.266 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\...stash_new.
• 07-01-2021 09:28:52.266 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\...stash_hec.
• 07-01-2021 09:28:52.266 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
• 07-01-2021 09:28:52.265 -0500 INFO TailingProcessor [66180 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME\var\run\splunk\search_telemetry\*search_telemetry.json.
• 07-01-2021 09:28:52.265 -0500 INFO TailingProcessor [66180 MainTailingThread] - TailWatcher initializing...

[b_chris21]

Had the same issue, here is what fixed it for me:

Downgraded my Splunk HF from 9.0.1 to the same version with the UFs that send data to it. There seems to be a conflict with the version mismatch, even though according to Splunk there a backwards compatibility for UFs.

Downgrade was uninstalling v9.0.1, installing v8.2.5 and unzipping an old good backup of my v8.2.5 /etc folder.

That made the trick.

Hope it helps. If yes, a Karma would be appreciated 🙂

Christos

[jdcabanglan]

I had encounter the same issue but as per checking my splunk version are both the same

[youngec]

For those who ugpraded to v9.x, this may be applicable:

https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/KnownIssues

2022-06-22SPL-226003 When forwarding from an 9.0 instance with useAck enabled, ingestion stops after some time with errors: "Invalid ACK received from indexer="

Workaround:
As a workaround, disable useAck in outputs.conf on the forwarder. After disabling, indexers start to ingest data.
If customers do need useACK to prevent data loss, disabling autoBatch in outputs.conf can remediate the issue too, but it impacts throughput - no worse than 8.x, but no improvement for 9.0.

[matt8679]

I had this issue too and noticed Splunk was falling behind when scanning large file before ingesting.

I ended up increasing the pipelines on the forwarders and the issue when away. Bumped to 3 where resources allowed.

[general]

parallelIngestionPipelines = 2

Also note, you will get this error if you have a source coming in with delayed logs. I think Splunk is alerting on this now so that is why you see the error with the updates. I still get this error on logs are are only coming in every couple of hours.

[verbal_666]

Whis is as you have 2 UF on same machine.

Maybe you should only increase the limits.conf,

[thruput]

maxKBps = <integer>
* The maximum speed, in kilobytes per second, that incoming data is
  processed through the thruput processor in the ingestion pipeline.
* To control the CPU load while indexing, use this setting to throttle
  the number of events this indexer processes to the rate (in
  kilobytes per second) that you specify.
* NOTE:
  * There is no guarantee that the thruput processor
    will always process less than the number of kilobytes per
    second that you specify with this setting. The status of
    earlier processing queues in the pipeline can cause
    temporary bursts of network activity that exceed what
    is configured in the setting.
  * The setting does not limit the amount of data that is
    written to the network from the tcpoutput processor, such
    as what happens when a universal forwarder sends data to
    an indexer.
  * The thruput processor applies the 'maxKBps' setting for each
    ingestion pipeline. If you configure multiple ingestion
    pipelines, the processor multiplies the 'maxKBps' value
    by the number of ingestion pipelines that you have
    configured.
  * For more information about multiple ingestion pipelines, see
    the 'parallelIngestionPipelines' setting in the
    server.conf.spec file.
* Default (Splunk Enterprise): 0 (unlimited)
* Default (Splunk Universal Forwarder): 256

Since by deault it send at 256Kb/s.

I set it to 2048 for many UFs which send much data.

You could also try a 0 to disable thruput control.

[linhmai_bne]

I got similar issue after upgrading 8.2.7. I have tried to set:

useAck=false

disable app Splunk...Forwarders

chown -R splunk:splunk /opt/splunk

but the problem is still there.

[tyates_ctm]

TL;DR: check `server` in `[tcpout:]` in `outputs.conf` of the server (not UFs)

I got this error after migrating onto bigger servers. The cause was the `server` attribute in the `[tcpout:]` stanza in `outputs.conf` on the various members of the cluster hadn't been updated. I have no idea why, but at some point over the past 5 years that same attribute on the UFs had been pointed at different DNS records, so the indexers were receiving the important data from across the estate.

Hope this helps someone.

[Gregski11]

we have a case open on this as well, I will keep you posted on the resolution

we see stuff like this, and then they just mysteriously go away and a few days later they return, we are on version 9.0.0

• Root Cause(s):
  • Events from tracker.log have not been seen for the last 1394 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.

[Gregski11]

 we are getting the same error on our Cluster Master and it's running version 9.0.0

• Root Cause(s):
  • Events from tracker.log are delayed for 44 seconds, which is more than the yellow threshold (15 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.

we also opened a support case with Splunk will keep you all up to date on how it unfolds

[jdcabanglan]

Did you fix the issue?

[Zacknoid]

Upgraded to version 9.0 facing similar issue : Root Cause(s) Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. did you find out any solution for this ?? 

Thanks 

[Gregski11]

---

@Zacknoid wrote:

Upgraded to version 9.0 facing similar issue : Root Cause(s) Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. did you find out any solution for this ?? 

Thanks 

---

no but after a day or two the problem just went away 

[Zacknoid]

Still looking for resolution, ingestion latency error 

[sombhtr239]

Anyone having solution please help

[sombhtr239]

I am also facing the same problem.  Server IOPS is 2000, still getting IOWAIT and ingesting latency error very frequently and then it goes away.

[Kathir]

Indicator 'ingestion_latency_gap_multiplier' exceeded configured value 

i am also getting 

[Marc_Williams]

So we upgraded to 8.2.2.1 and are still getting the error. However it is a bit different than before.

• Events from tracker.log have not been seen for the last 1395 seconds, which is more than the red threshold (210 seconds). This typically occurs when indexing or forwarding are falling behind or are blocked.

[salbro]

Also seeing this issue after moving from 8.1.2 to 8.2.2. We are using older hardware, but this makes me think it is not necessarily related. It comes and goes throughout the day.

[apietersen]

Same here, on Splunk Ent. v8.2.2

[Funderburg78]

I am also having this issue but only one one of 6 splunk servers.  The other Splunk servers do not have a tracker.log.  This log is not listed in: https://docs.splunk.com/Documentation/Splunk/8.2.2/Troubleshooting/Enabledebuglogging#log-local.cfg as a splunk log so I wonder if it has something to be done with the upgrade.  

It has been 1 week since my upgrade and this is the only server complaining.  Would really like to know what this log is and why it is having issues.  I checked file permissions and it is the same as the other logs.... 

This log is in /var/spool/splunk and is a default to be monitored in the /splunk/etc/system/default/inputs.con and is listed as a latency tracker.  of my 6 servers only the search head running ES even has this log in the director y