Why is Splunk not getting all Windows security events from some Active Directory servers?

[snix]

I have the universal forwarder installed on three Active Directory servers and I have a dashboard with a panel that shows when there is a failure under "Microsoft Windows security auditing" for a specific set of event ID's.

I also have a similar setup using another syslog solution that we were using before we bought Splunk, and during the transition of moving to Splunk, we are using both in tandem to make sure Splunk sees everything the previous solution does.

Everything has been great so far with one or two small exceptions and one of them popped up today. I have seen it before and now I just want to resolve it. So what happens is I will see a specific security event ID 4662 show up on the dashboard in Splunk and it shows what user caused the event to occur, but I only see it come from... lets call it "ADServer1," but I never see any similar events pop up from either "ADServer2" or "ADServer3". However, when I log into the older syslog solution, I can see similar events coming from servers 2 and 3. So I logged onto all three servers to verify this in their event viewers and it all matched up with what I see in the older solution. For some reason, Splunk only reports the events happening on the first server, but not the other two servers even though it does happen on all three.

I know it is also not my search string I use in the dashboard panel as I will just do a very basic search just for "EventCode=4662" and still only see results from the first server. I also tried to look at the config files of each server and although I am by far no expert I was not able to find anything that was different or that could some how allow most security logs to get forwarded but not some.

Not sure what to do at this point to track down the issue.

[Richfez]

This behavior can be overridden by something like the following stanza:

[WinEventLog://Security]
blacklist1 = ""
blacklist2 = ""

Put that in Splunk_TA_windows/local/inputs.conf, either on your Deployment Server (in $SPLUNK_HOME/etc/deployment-apps/) if you are using the DS, or directly on each system (in $SPLUNK_HOME/etc/apps/) if not.

Note these will stick around after upgrades, so don't forget about them.

[dflodstrom · Accepted solution]

If you don't have any local configurations it could be that this particular event code is filtered out by default. In Splunk_TA_windows/default/inputs.conf you will see:

[WinEventLog://Security]
disabled = 1
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
index = wineventlog
renderXml=false

The parameter " blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)" " is filtering out these events by default. You can comment this out of default/inputs.conf but it is recommended that you override that setting in local/inputs.conf either in that TA or a custom TA. I imagine you're enabling these inputs somewhere, put it there.

[Richfez]

Specifically, to "comment those out", place a stanza like

[WinEventLog://Security]
blacklist1 = ""
blacklist2 = ""

In the local/inputs.conf for the app involved.

[snix]

I don't use another TA I was just planning on using the default Forwarder install to pull just the event logs. Am I missing something if I take it off? I don't currently plan on collecting performance logs so I figure the default Forwarder install can do that and I can just toss the Windows TA.

[dflodstrom]

If you install the universal forwarder on a Windows server and don't configure any inputs or TAs it will not collect Windows events by default. If you've chosen to collect these logs as a part of your installation and you're not installing a TA then maybe it is putting something in /local/inputs.conf on your forwarder installation.

[snix]

Your a genus! That was it, I was scratching my head thinking what is Splunk_TA_windows and then I remember when I was testing I was told to try Splunk's Windows monitoring app. Well needless to say after a week or so of trying to mess with it and part of that was the Splunk_TA_windows app on the AD servers I ended up giving up after trying to configure it I some how hosed that Splunk install. So the Splunk_TA_windows app must still be on both servers.

Just to verify I can just delete out Splunk_TA_windows Folder and that should remove the app itself correct? I have no plans on using the Spunk Windows app so I don't see why I would need it at this point.

[dflodstrom]

If you're using another TA to gather Windows events from your AD servers then you don't need Splunk_TA_windows and yes, you can delete it just by removing that folder from the apps directory on a server where it is installed. If you're using a deployment server you may want to select 'uninstall' from the forwarder management dashboards in the web GUI.

[jnussbaum_splun]

Can you verify that you're receiving events from the two AD servers in question? If yes, is the data that's coming showing event codes? Can you try to search: host=ADServer1 OR host=ADServer2 OR host=ADServer3 4662and see if the string matches?

[snix]

I can verify both AD servers that do not report this specific event do report event logs in general and that includes other security logs just not the ones related to event ID 4662. There may be other event ID's they are not being reported but I pulled tones of related logs from both servers in just the last couple of days related to security and user logins. It just seems to be just be this one event or hand full of events that are not pulling and 4662 is one of them that I found.