topic Re: Delay during log ingestion from Azure in Getting Data In

Delay during log ingestion from Azure

antnovo — Tue, 26 Feb 2019 21:11:10 GMT

Hello, have a question regarding log ingestion from Azure. At the moment, im using REST API to onboard logs to the on premise Heavy Forwarder which sends data to indexes located on splunkcloud.

For some reason there's a huge delay between event indexing and event creation time, still receiving logs that are 3 months old and new logs are getting delayed. What can be a reason for such a delay? Is it a normal behavior during Azure and Splunk integration?

Thank you in advance.

Re: Delay during log ingestion from Azure

integratorz — Thu, 28 Feb 2019 19:04:36 GMT

@antnovo I would check the throughput restriction Splunk has by default. It throttles how much data splunk can send to 256kbps. This is done in limits.conf:

[thruput]
# setting this to 0 means makes it unlimited (be careful as a single forwarder can overwhelm an indexer)
maxKBps = 0

Re: Delay during log ingestion from Azure

antnovo — Fri, 01 Mar 2019 17:13:42 GMT

Hi,

Thank you for suggestion, checked limits.conf and it was already set to 0. Could it be related to usage of REST? Haven't found a single issue like this in Splunk answers related to log injection via Splunk add-on for Microsoft Cloud Services.
I'm considering to switch from REST to direct integration of Splunk and Azure via App but not sure if it will solve the problem.

Have a great weekend

Re: Delay during log ingestion from Azure

integratorz — Fri, 01 Mar 2019 17:29:11 GMT

What is the interval you a querying the API at?

Re: Delay during log ingestion from Azure

antnovo — Wed, 06 Mar 2019 10:18:46 GMT

Hi, i was querying the API at 1h interval, changed it this morning to 5 minutes.

Thanks

Re: Delay during log ingestion from Azure

Fulvio — Thu, 15 May 2025 10:14:16 GMT

I know this thread is old, but this information may still help.

As specified in Microsoft Learn portal, "Microsoft doesn't guarantee a specific time after an event occurs for the corresponding audit record to be returned in the results of an audit log search. For core services (such as Exchange, SharePoint, OneDrive, and Teams), audit record availability is typically 60 to 90 minutes after an event occurs. For other services, audit record availability might be longer. However, some issues that are unavoidable (such as a server outage) might occur outside of the audit service that delays the availability of audit records. For this reason, Microsoft doesn't commit to a specific time."

Re: Delay during log ingestion from Azure

isoutamo — Thu, 15 May 2025 10:23:22 GMT

This same information has said some other places in MS documentation too. Basically (almost) all logs have some delays when you try to get those via Azure own functionality. But if you install UF then you get those immediately.