All Apps and Add-ons

Windows Event Logging delayed

hrawat_splunk
Splunk Employee
Splunk Employee

Windows event logs are delayed for days. Latency varies, at times it's few minutes or several thousands seconds. Confirmed that none of the pipeline queues are blocked.

0 Karma
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

NOT FOR 8.1.x and 8.2.x versions as these versions lost DEBUG logs.

There are several reasons for delayed indexing of windows events.
Turn on DEBUG. Use following steps.
1. #Set in splunk_home\etc\log-cmdline-local.cfg:
category.splunk-winevtlog=DEBUG
2. #And also set in splunk_home\etc\log-local.cfg:
category.ExecProcessor=DEBUG
3. #Increase number of backup files for splunkd.log, set in splunk_home\etc\log-local.cfg:
appender.A1.maxBackupIndex=50
4. Restart splunk
5. Once problem is re-created take diag
6. Delete splunk_home\etc\log-cmdline-local.cfg and splunk_home\etc\log-local.cfg ( turns off log after step #7)
7. Restart splunk .

If you find excessive logging for the windows channel you are interested. For example

08-02-2019 00:00:01.030 -0100 INFO ExecProcessor - message from ""D:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::getEventsNew: Failed to open publisher metadata provider '' for event log, channel=''

Then the root case of the delayed indexing is, there is an attempt( eventually failed) to try to open manifest file of given provider, for each event fetched. It introduces excessive I/O operation that will be slowing down indexing thruput for the channel in question.
Once manifest file is Installed, it's cached and you should see significantly improved indexing thruput for that given channel.

View solution in original post

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Unfortunately all 8.1.x and 8.2.x versions are not providing splunk-winevtlog DEBUG logs, as a result this post is not able to deflect the new cases against 8.1.x and 8.2.x. 

In order to troubleshoot WEC related event log delay , follow one of the approach.

1. Downgrade to latest 8.0.x.

2. Stop splunk, Copy 8.0.x $SPLUNK_HOME\bin\splunk-winevtlog.exe, enable logging as suggested and start splunk

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

NOT FOR 8.1.x and 8.2.x versions as these versions lost DEBUG logs.

There are several reasons for delayed indexing of windows events.
Turn on DEBUG. Use following steps.
1. #Set in splunk_home\etc\log-cmdline-local.cfg:
category.splunk-winevtlog=DEBUG
2. #And also set in splunk_home\etc\log-local.cfg:
category.ExecProcessor=DEBUG
3. #Increase number of backup files for splunkd.log, set in splunk_home\etc\log-local.cfg:
appender.A1.maxBackupIndex=50
4. Restart splunk
5. Once problem is re-created take diag
6. Delete splunk_home\etc\log-cmdline-local.cfg and splunk_home\etc\log-local.cfg ( turns off log after step #7)
7. Restart splunk .

If you find excessive logging for the windows channel you are interested. For example

08-02-2019 00:00:01.030 -0100 INFO ExecProcessor - message from ""D:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::getEventsNew: Failed to open publisher metadata provider '' for event log, channel=''

Then the root case of the delayed indexing is, there is an attempt( eventually failed) to try to open manifest file of given provider, for each event fetched. It introduces excessive I/O operation that will be slowing down indexing thruput for the channel in question.
Once manifest file is Installed, it's cached and you should see significantly improved indexing thruput for that given channel.

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Recently with one customer case found that all 8.1.x and 8.2.x versions are not providing splunk-winevtlog DEBUG logs, as a result this post was not able to deflect the new case against 8.1.x and 8.2.x. 

In order to troubleshoot WEC related event log delay , follow one of the approach.

1. Downgrade to latest 8.0.x.

2. Stop splunk, Copy 8.0.x $SPLUNK_HOME\bin\splunk-winevtlog.exe, enable logging as suggested and start splunk

0 Karma

dbot2001
Path Finder

Are these DEBUG settings to be done on the UF on the windows host?

How does the manifest file get installed to resolve this?

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee
0 Karma

dstaulcu
Builder

Good technique to know. Sounds like that condition would be faster to diagnose with powershell get-winevent cmdlet. Also sounds like a useful warning for winevent log component to write to splunkd in future versions. Eg. warn when getevent feature takes more than, say 10 seconds, to return a result.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!