Getting Data In

Windows Security Events only being forwarded for a short time after restarting universal forwarder

mccartneyc
Path Finder

Hi everyone, I have about 20 windows servers and 30 linux servers, all with universal forwarders installed and configured. We've recently found out that the windows servers are not forwarding their security events properly.

Currently the windows servers monitor System, Application, and Security events. All of the System and Application events are being forwarded properly and never seem to drop. Our issue is with the Security events.

After starting the universal forwarders on all of the windows server, the Security events get forwarded to the indexer cluster and are searchable. They will continue forwarding the Security events for a couple of hours and then randomly stop. Once they stop sending Security events, they will not send them anymore until the forwarder service is restarted and then they will send for a couple of hours and stop again. Longest we've waited to see if they eventually send is 1 week, but they do not send until a restart.

This is only for the Security events on Windows servers. All other inputs on windows servers are forwarding without issue and after restarting the forwarder service the security events will forward for a couple of hours and then stop until the service is restarted.

Nothing is showing in the logs. All of the internal splunk logs show the Security events being sent and then it just stops without any notice or error and continues sending other inputs.

Any assistance with this is appreciated, thanks!

0 Karma
1 Solution

mccartneyc
Path Finder

Haha...so I fixed my issue. Turns out when the start_from setting in input is set to newest, it will only run once upon splunk starting and will only get events that occurred when splunk started and older events. It will not continue collecting events. When it's done collected events prior to startup it will stop collecting until splunk is restarted.

The thought was if it starts from newest, it would forward all new events in real time and continue grabbing the old events. Missed the last like 5 words in Splunk description of the setting. Just deleted the start_from line from the confs, splunk will default to oldest. Currently have been monitoring it for about 2 hours and since I removed that line, splunk is forwarding all WinEventLog:Security sourcetype events as they come in.

View solution in original post

0 Karma

mccartneyc
Path Finder

Haha...so I fixed my issue. Turns out when the start_from setting in input is set to newest, it will only run once upon splunk starting and will only get events that occurred when splunk started and older events. It will not continue collecting events. When it's done collected events prior to startup it will stop collecting until splunk is restarted.

The thought was if it starts from newest, it would forward all new events in real time and continue grabbing the old events. Missed the last like 5 words in Splunk description of the setting. Just deleted the start_from line from the confs, splunk will default to oldest. Currently have been monitoring it for about 2 hours and since I removed that line, splunk is forwarding all WinEventLog:Security sourcetype events as they come in.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...