One of the issues I recently dealt with was the delay in sending security channel logs in Active Directory, which I finally resolved after a few days. Here are the steps I took to fix the problem: I investigated the queue issue in different pipelines. This link explains in detail how to identify and fix queue problems to reduce delays: index=_internal host=* blocked=true This way, you can check whether the issue is with the universal forwarder, the heavy forwarder, or a higher tier. I experienced this issue with both UF and HF. I increased the queue size and added the following parameter along with the queue adjustment: /etc/system/local/server.conf parallelIngestionPipelines=2 https://conf.splunk.com/files/2019/slides/FN1570.pdf To adjust the max speed rate in the ingestion pipeline, I modified the following parameter in limits.conf: [thruput] maxKBps = 0 https://community.splunk.com/t5/Getting-Data-In/How-can-we-improve-universal-forwarder-performance/m-p/276195 The final and most effective step was changing the following parameter in UF’s inputs.conf: use_old_eventlog_api=true If you have added the parameter evt_resolve_ad_obj=true to translate SID/GUID and it cannot perform the translation, it will pass the task to the next domain controller. It waits for a response before proceeding, which can cause delays. To fix this, I added: evt_dc_name=localhost By implementing the above steps, logs were successfully received and indexed in real-time. Thank you for taking the time to read this. I hope it helps you resolve similar issues.
... View more
