Windows Security Events only being forwarded for a short time after restarting universal forwarder

[mccartneyc]

Hi everyone, I have about 20 windows servers and 30 linux servers, all with universal forwarders installed and configured. We've recently found out that the windows servers are not forwarding their security events properly.

Currently the windows servers monitor System, Application, and Security events. All of the System and Application events are being forwarded properly and never seem to drop. Our issue is with the Security events.

After starting the universal forwarders on all of the windows server, the Security events get forwarded to the indexer cluster and are searchable. They will continue forwarding the Security events for a couple of hours and then randomly stop. Once they stop sending Security events, they will not send them anymore until the forwarder service is restarted and then they will send for a couple of hours and stop again. Longest we've waited to see if they eventually send is 1 week, but they do not send until a restart.

This is only for the Security events on Windows servers. All other inputs on windows servers are forwarding without issue and after restarting the forwarder service the security events will forward for a couple of hours and then stop until the service is restarted.

Nothing is showing in the logs. All of the internal splunk logs show the Security events being sent and then it just stops without any notice or error and continues sending other inputs.

Any assistance with this is appreciated, thanks!

[mccartneyc · Accepted solution]

Haha...so I fixed my issue. Turns out when the start_from setting in input is set to newest, it will only run once upon splunk starting and will only get events that occurred when splunk started and older events. It will not continue collecting events. When it's done collected events prior to startup it will stop collecting until splunk is restarted.

The thought was if it starts from newest, it would forward all new events in real time and continue grabbing the old events. Missed the last like 5 words in Splunk description of the setting. Just deleted the start_from line from the confs, splunk will default to oldest. Currently have been monitoring it for about 2 hours and since I removed that line, splunk is forwarding all WinEventLog:Security sourcetype events as they come in.