Inconsistent Windows Security Events

[gitau_gm]

I am observing inconsistent forwarding of Windows Security Event ID 4624 (Successful Logon) from multiple Windows hosts running Splunk Universal Forwarder, despite the hosts having identical configurations.

Some devices forward Event ID 4624 reliably, while others do not forward 4624 at all. This inconsistency persists even though configuration, versions, and OS‑level settings are the same.

[PickleRick]

The usual debugging steps/questions (adding to what @livehybrid already asked)

1. Are 4624 the only events which are not forwarder?

2. Are they not forwarded at all from selected clients or just some of them?

3. Does the splunkd.log show anything interesting for those clients?

4. Are you absolutely sure you have the same effective configuration? (I'm not talking only about what you're pushing but about the effective config merged from all apps, system/local and so on).

[livehybrid]

Hi @gitau_gm 

For the instance not sending Windows Security Event ID 4624, does it sometimes send but not others? 

Are you able to confirm that both instances are running Splunk as the same user with the same permissions? Are you able to see Windows Security Event ID 4624 when checking directly on the instance which is not sending these events? 

How are you managing these instances? Is the content pushed out via a Deployment Server?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[gitau_gm]

@livehybrid  The 4624s are the ones that made me realize there was an issue

These are the events being forwarded by the "good" devices (forwarding 4624s) 

Authentication / Logon Activity
User Rights & Privileges
Service Control Manager
System & Service Lifecycle
Credential Validation (Kerberos/NTLM)
IPsec / Network Security

These are the only events being forwarded by the "other" devices

Service Control Manager
System & Service Lifecycle

[PickleRick]

If your whole logs are not being forwarded, not just selected events, it suggests some permission issue. Probably either the user your forwarder runs under is not a member of the right group or you have some different policy applied on those UFs which doesn't grant sufficient rights to event logs. (or someone edited ACLs on logs only on some endpoints). That's something I would be troubleshooting with my local windows/AD guru.

[gitau_gm]

Some devices are forwarding all windows events and some are not. Checked confs and they are identical in both instances.

[PickleRick]

As I said - it's not about the difference in UF's configuration. It's probably at the OS-level. The user under which the UF's splunkd.exe process is being run does not have the right to read the event logs. After you restart the UF you should get messages (I don't remember if those are warnings or errors) about the input not being able to subscribe to specific event logs.

[gitau_gm]

Hey PickeRick

In both instances, the messages are consistent

Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from 'C:\Program Files\SplunkUniversalForwarder\splunkforwarder-10.2.2--windows-x64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... SplunkForwarder: Starting (pid 32356) Done

[PickleRick]

But that is not what I was talking about. What you're pasting is the messages Splunk service emits on stdout or stderr when you're starting it. I was talking about the messages which the forwarder writes to splunkd.log (and susbsequently picks up with monitor input and sends to the indexers). Those messages might contain a hint about why the event logs are not read.

As I wrote before, the probable cause is that the splunkd.exe process is not able to subscribe to the event log channel(s). It has nothing to do with the forwarder's config itself (the files in $SPLUNK_HOME\etc), but it depends on the user the splunkd.exe is run from.

It's an issue with the OS configuration regarding this user and is a fairly common thing - the user which the Splunk Universal Forwarder service is running as must have appropriate permissions granted on the operating system level (either manually or via GPO) to be able to access some of the event log channels.

[gitau_gm]

Hey @livehybrid we are migrated to splunk cloud a couple of months ago so we have ufs sending directly to the cloud. All configs are the same, and yes, I do see security events from the endpoints directly for both instances. We have a DS pushing content.