So we just updated to 8.2.1 and we are now getting an Ingestion Latency error…
How do we correct it? Here is what the link says and then we have an option to view the last 50 messages...
Ingestion Latency
Here are some examples of what is shown as the messages:
07-01-2021 09:28:52.269 -0500 INFO TailingProcessor [66180 MainTailingThread] - Adding watch on path: C:\Program Files\CrushFTP9\CrushFTP.log.
I got similar issue after upgrading 8.2.7. I have tried to set:
useAck=false
disable app Splunk...Forwarders
chown -R splunk:splunk /opt/splunk
but the problem is still there.
TL;DR: check `server` in `[tcpout:]` in `outputs.conf` of the server (not UFs)
I got this error after migrating onto bigger servers. The cause was the `server` attribute in the `[tcpout:]` stanza in `outputs.conf` on the various members of the cluster hadn't been updated. I have no idea why, but at some point over the past 5 years that same attribute on the UFs had been pointed at different DNS records, so the indexers were receiving the important data from across the estate.
Hope this helps someone.
we have a case open on this as well, I will keep you posted on the resolution
we see stuff like this, and then they just mysteriously go away and a few days later they return, we are on version 9.0.0
we are getting the same error on our Cluster Master and it's running version 9.0.0
we also opened a support case with Splunk will keep you all up to date on how it unfolds
Did you fix the issue?
Upgraded to version 9.0 facing similar issue : Root Cause(s) Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. did you find out any solution for this ??
Thanks
@Zacknoid wrote:Upgraded to version 9.0 facing similar issue : Root Cause(s) Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. did you find out any solution for this ??
Thanks
no but after a day or two the problem just went away
Still looking for resolution, ingestion latency error
Anyone having solution please help
I am also facing the same problem. Server IOPS is 2000, still getting IOWAIT and ingesting latency error very frequently and then it goes away.
Indicator 'ingestion_latency_gap_multiplier' exceeded configured value
i am also getting
So we upgraded to 8.2.2.1 and are still getting the error. However it is a bit different than before.
Also seeing this issue after moving from 8.1.2 to 8.2.2. We are using older hardware, but this makes me think it is not necessarily related. It comes and goes throughout the day.
Same here, on Splunk Ent. v8.2.2
I am also having this issue but only one one of 6 splunk servers. The other Splunk servers do not have a tracker.log. This log is not listed in: https://docs.splunk.com/Documentation/Splunk/8.2.2/Troubleshooting/Enabledebuglogging#log-local.cfg as a splunk log so I wonder if it has something to be done with the upgrade.
It has been 1 week since my upgrade and this is the only server complaining. Would really like to know what this log is and why it is having issues. I checked file permissions and it is the same as the other logs....
This log is in /var/spool/splunk and is a default to be monitored in the /splunk/etc/system/default/inputs.con and is listed as a latency tracker. of my 6 servers only the search head running ES even has this log in the director y
I am going to reach out to support when I get a chance and will update here when I have found a solution/workaround of some sort. My OS is Linux and the log path/permission looks fine from my perspective as well. We upgraded over a month ago and this issue persists but only on our indexer. Our heavy forwarders are not affected by this.
Have you heard back from support regarding this issue? We have been running on 8.2.2 for several weeks without issue, but today noticed this on one of the search heads within the SHC.
My apologies, we actually redeployed for a separate issue we were facing so I never did contact them on this.
I am having this issue as well. Would appreciate any information you've been able to dig up.
Hi Marc,
We are facing the same issue after 8.2.1 upgrade
Have you already found a solution?
Greetings,
Justyna