topic Re: How to stop getting duplicate events from WindowsUpdateLog? in Getting Data In

How to stop getting duplicate events from WindowsUpdateLog?

hettervik — Mon, 18 Nov 2019 11:57:34 GMT

Hi. I have a problem that I'm getting duplicate events from WindowsUpdate.log. I'm pretty sure it's related to the following message I'm getting in the internal logs Logfile truncated while open, original pathname="C:/winupdate/blabla", will begin reading from start. Though, if the logfile really was truncated (all the data in the file deleted), then I wouldn't get duplicate events. It seems to me that Splunk only "thinks" that the log file was truncated, but in reality it's not.

Can anyone explain to me how this "truncating" of the WindowsUpdateLog works, and why it can cause duplicate events in Splunk? Also, if someone could offer a possible solution, that would be nice as well!

Re: How to stop getting duplicate events from WindowsUpdateLog?

DavidHourani — Mon, 18 Nov 2019 12:06:07 GMT

hi @hettervi,

This is an informational message, isn't it ? Doesn't seem like an error to me. It only means that the log file got rotated, if you're getting duplicates it could be because the file is generated with the same data again.

Is there any timestamp in the file or anything that makes the events unique ? You could try to play around with initCrcLength and crcSaltto avoid having the file indexed over and over

Re: How to stop getting duplicate events from WindowsUpdateLog?

hettervik — Tue, 19 Nov 2019 08:30:53 GMT

Yes. It's not an error message per se. It doesn't necessarily mean that the windowsupdatelog got rotated, and that's the problem I think. For some reason, instead of rotating the logs in a normal fashion, Windows truncate the log file and starts writing it over again, if I understand correctly, but I'm not sure. Though it doesn't make any sense that Windows would generate the same file over again with the same data.

I have timestamps, and they are not unique. That is, I'm positive that the events I'm getting are in fact duplicate events from the same machine. Same timestamp, same everything, except indextime of course.

Re: How to stop getting duplicate events from WindowsUpdateLog?

DavidHourani — Tue, 19 Nov 2019 08:39:48 GMT

so from what you're saying it could be that since Windows is modifying something in the header of the file the rest of the file is getting indexes all over again.

Could you please share your inputs configuration ?

Re: How to stop getting duplicate events from WindowsUpdateLog?

hettervik — Tue, 19 Nov 2019 20:12:00 GMT

I don't have the input configuration available right now unfortunately, but it's straight from the official Windows TA from Splunk. Is uses a normal monitoring stanza for tailing the log. I think the path is C:\Windows\Logs\WindowsUpdate.

Re: How to stop getting duplicate events from WindowsUpdateLog?

hettervik — Mon, 25 Nov 2019 14:13:08 GMT

For the record I've also tried using the setting ignoreOlderThan=1h, but it has no effect. I will try using another inputs.conf setting followTail=1. It's kind of a hacky solution, but if it stops duplicate entries I'm happy.

Re: How to stop getting duplicate events from WindowsUpdateLog?

hettervik — Fri, 02 Sep 2022 13:51:26 GMT

Seems like this problem has been addressed in newer versions of the supported Splunk Windows TA.

https://docs.splunk.com/Documentation/AddOns/released/Windows/Configuration#Configure_Windows_Update_Logs_in_inputs.conf