Delayed log ingestion

[cymondcuba]

Hi Splunk,

Having a problem with one of our ingestion in splunk. The logs are delayed and cant seem to find the cause of the ingestion issue. Could someone help us what would be the troubleshooting to be done? and what might be causing the issue as the logs are delayed for a day.

Thank you,

[gjanders]

There are various places this could happen, at the indexer level you should be looking at your monitoring console , are the event pipelines blocked?

At the forwarder level, you can check this via the splunkd.log file which will advise if the throttling limit for the forwarder has been reached or not, and if you are not just reaching a throttle limit which you can change in limits.conf you could then look into your metrics.log on the forwarder to see if limits are reached there.

Are your forwarders connecting directly to indexers? If not you can use the monitoring console to check the next heavy forwarder in the chain before it gets to the indexer if that is the case.

The Splunk conf 2017 had a few sessions around troubleshooting which might help here, note I've added some filters there you may wish to turn them off / change them to find more sessions...