Why are Windows Event Logs sent and indexed every half hour?

[mmccul]

Our domain controllers were resending the entire Windows EventLog every 30 minutes. No duplicate inputs entries. No duplicate outputs entries. Using a search like

host=foohost index=bar_index earliest=-1h | 
convert ctime(_indextime) | 
convert ctime(_time) | 
stats count list(host) list(splunk_server) list(_time) list(_indextime) by _raw 

would display multiple indexings of the exact same _raw message, even the timestamp. Adjusting the earliest parameter showed that it happened every 30 minutes.

We encountered this on Splunk 6.5.1

We tried a fresh re-install of the forwarder, no change.

We tried inspecting the checkpoint file, but it wasn't corrupt or anything.

[mmccul · Accepted solution]

After much debugging with Splunk support, we determined that having

start_from=newest

in our configs was a primary cause of the behavior. Disabling that directive on the inputs, going back to the default

start_from = oldest

eliminated the issue immediately. Hopefully this helps others.

[abalogh_splunk]

Hi,
- Is the sourcetype the same? WMI enabled? Possible some other host is doing WMI collection? sourcetype would tell you that.
- If you disable inputs.conf will you still get logs?
- Check metrics.logs
- How was the UF installed? CLI installation with WINEVENTLOG_SEC_ENABLE=0 WINEVENTLOG_SYS_ENABLE=0 ?

I hope this help you debugging the issue!