There are several reasons for delayed indexing of windows events.
Turn on DEBUG. Use following steps.
1. #Set in splunkhome\etc\log-cmdline-local.cfg:
2. #And also set in splunkhome\etc\log-local.cfg:
3. #Increase number of backup files for splunkd.log, set in splunkhome\etc\log-local.cfg:
4. Restart splunk
5. Once problem is re-created take diag
6. Delete splunkhome\etc\log-cmdline-local.cfg and splunk_home\etc\log-local.cfg ( turns off log after step #7)
7. Restart splunk .
If you find excessive logging for the windows channel you are interested. For example
08-02-2019 00:00:01.030 -0100 INFO ExecProcessor - message from ""D:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::getEventsNew: Failed to open publisher metadata provider '' for event log, channel=''
Then the root case of the delayed indexing is, there is an attempt( eventually failed) to try to open manifest file of given provider, for each event fetched. It introduces excessive I/O operation that will be slowing down indexing thruput for the channel in question.
Once manifest file is Installed, it's cached and you should see significantly improved indexing thruput for that given channel.
Good technique to know. Sounds like that condition would be faster to diagnose with powershell get-winevent cmdlet. Also sounds like a useful warning for winevent log component to write to splunkd in future versions. Eg. warn when getevent feature takes more than, say 10 seconds, to return a result.