Windows Event Logs Not Forwarding Security Events to Splunk

[abdallah_bakr]

 I've got Splunk Universal Forwarder up and running on my DC-01, and it's set to forward all Windows event logs to Splunk. But there's a catch - it's not forwarding the Security events for some reason!

Interestingly, when I installed the UF on a regular Windows PC, everything worked like a charm, and all event types, including Security events, were forwarded without a hitch. 

I've done my fair share of digging through documentation and troubleshooting cases, but I'm still at a loss. It feels like it might be a permissions or rights issue, but I can't seem to find the root cause.

If any of you have encountered a similar issue or have any insights, I'd be incredibly grateful for your help and ideas.

Thank you in advance for any guidance you can provide!

[gcusello]

Hi @abdallah_bakr,

did you check is in the inputs.conf file of the Splunk_TA-Windows App, the Windows Security input of that server is enabled or not?

Ciao.

Giuseppe

[abdallah_bakr]

Hi @gcusello 

Of course.

I am also using a deployment server to deploy Splunk_TA_windows for all Windows machines so that the configurations are the same for all Windows machines.

And as I previously mentioned, when I installed the UF on a regular Windows PC, everything worked like a charm, and all event types, including Security events, were forwarded without a hitch. So I am only facing this issue only on DC-01.

[isoutamo]

Are the other DCs working as expected?

[abdallah_bakr]

We have 4 DCs, It's not working on any of them but it works on other Windows servers and workstations.

[isoutamo]

Then it’s definitely some access issue with DC. Quite probably some user base (in which user your UF is running) or GPO changes is needed.

[abdallah_bakr]

The UF runs under Local System account.

[isoutamo]

That’s the reason why it’s not working. To monitor AD you must use domain account which have access to AD. See more from docs https://docs.splunk.com/Documentation/Splunk/9.1.0/Data/MonitorActiveDirectory

[abdallah_bakr]

I went through the article. It talks about installing Splunk Enterprise on Windows not Splunk Universal Forwarder and by the way I have installed the UF on DCs tens of times using the same way and running under the Local System account (which is the default installation) and never encountered the issue. I have this also running on my LAB AD environment without any issues.

[isoutamo]

As @PickleRick also said, this depends which kind of access you have in your ad environment. If I have understood right (I’m not a windows guru), it’s best practices to deny additional access from system local to ad environment. There are other roles defined for managing ad. I suppose that there are too many ad environments where these hardenings haven’t implemented. 
I said that in security point of view it’s much better to use MSA than modify your current GPO to allow access to your system local account. 
You should find from even logs that there are some denied attempts to access it, when you are using domain admin (or similar) role.

[PickleRick]

It is most probably a permissons issue. Unfortunately, the ACLs for eventlog channels are buried somewhere in the registry so you have to dig them up and check if your user is included. And yes, it's not that easy to get Security logs on Windows Server.

[abdallah_bakr]

That's strange because I have installed tons of Universal Forwarder on Windows servers/machines and I have never faced this issue before.

[PickleRick]

Yes, but DCs most often have different GPOs assigned to them and the user you're running your forwarder with might not be given proper permissions for that particular log. That might be a case of some group membership but I'm not 100% sure here.

[abdallah_bakr]

The UF runs under Local System account

[PickleRick]

I'm not very strong on windows magic but I'd say that either run UF under managed service account (as @isoutamo hinted) or push GPO to your DCs allowing access to the Security log for Local System. The former being probably the preferred solution since there might be several different things running as Local System and yoh might not want them all having access to Security logs.