Why is Windows Event Log Whitelist not working?

[shocko]

Running a Windows 2012 R2 DHCP Server with UF 9.0.1 and Splunk Enterprise 8.0.5. My inputs at the UF look like this:

[default]
index = windowsdhcp
_TCP_ROUTING = prod

[WinEventLog://System]
start_from = oldest
disabled = 0
current_only = 0
whitelist1 = SourceName="DhcpServer"
whitelist2 = SourceName="Dhcp-Server"

[WinEventLog://DHCPAdminEvents]
start_from = oldest
disabled = 0

My issue is that the whitelisted events in the 1st stanza are not getting processed to the indexer. If I review the XML of the events in the Windows Event Viewer:

These events are collected and indexed:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-DHCP-Server" Guid="{6D64F02C-A125-4DAC-9A01-F0555B41CA84}" /> 
  <EventID>20251</EventID> 
  <Version>0</Version> 
  <Level>4</Level> 
  <Task>121</Task> 
  <Opcode>106</Opcode> 
  <Keywords>0x2000000000000000</Keywords> 
  <TimeCreated SystemTime="2022-10-29T12:25:40.655052000Z" /> 
  <EventRecordID>161</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="3884" ThreadID="4472" /> 
  <Channel>DhcpAdminEvents</Channel> 
  <Computer>dhcp-srv-a.mydomain.com</Computer> 
  <Security UserID="S-1-5-20" /> 
  </System>
- <EventData>
  <Data Name="Server">dhcp-srv-b.mydomain.com</Data> 
  <Data Name="RelationName">dhcp-srv-b.mydomain.com-dhcp-srv-a.mydomain.com</Data> 
  <Data Name="OldState">COMMUNICATION_INT</Data> 
  <Data Name="NewState">NORMAL</Data> 
  </EventData>
  </Event>

These events do not get captured (Note: event is in classic format):

Log Name:      System
Source:        Microsoft-Windows-DHCP-Server
Date:          14/11/2022 23:11:37
Event ID:      1376
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dhcp-srv-a.mydomain.com
Description:
IP address range of scope 10.119.6.0 is 89 percent full with only 6 IP addresses available.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DHCP-Server" Guid="{6D64F02C-A125-4DAC-9A01-F0555B41CA84}" EventSourceName="DhcpServer" />
    <EventID Qualifiers="0">1376</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2022-11-14T23:11:37.000000000Z" />
    <EventRecordID>87097</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>dhcp-srv-a.mydomain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>10.119.6.0</Data>
    <Data>89</Data>
    <Data>6</Data>
  </EventData>
</Event>

I can't see why it is not collecting the second event via the 1st stanza?

[shocko]

The plot thickens! So the sourcename was actually Microsoft-Windows-DHCP-Server. This is as per the copy/paste to notepad of the XML view in Widows event veiwer and shown above as

Log Name:      System
Source:        Microsoft-Windows-DHCP-Server
Date:          14/11/2022 23:11:37
Event ID:      1376
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dhcp-srv-a.mydomain.com
Description:
IP address range of scope 10.119.6.0 is 89 percent full with only 6 IP addresses available.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DHCP-Server" Guid="{6D64F02C-A125-4DAC-9A01-F0555B41CA84}" EventSourceName="DhcpServer" />
    <EventID Qualifiers="0">1376</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2022-11-14T23:11:37.000000000Z" />
    <EventRecordID>87097</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>dhcp-srv-a.mydomain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>10.119.6.0</Data>
    <Data>89</Data>
    <Data>6</Data>
  </EventData>
</Event>

I updated my input stanza as follows and now working :

[WinEventLog://System]
start_from = oldest
disabled = 0
current_only = 0 
whitelist1 = SourceName="Microsoft-Windows-DHCP-Server"

I basically pushed all events from the System event log to my testSplunk instance and saw what SourceName it pulled out of the classic event.

That said, I am now still getting ALL events frm the System log! My entire inputs.conf file is as follows

[default]
index = windowsdhcp
_TCP_ROUTING = ssl-production

[WinEventLog://System]
start_from = oldest
disabled = 0
current_only = 0 
whitelist = SourceName="Microsoft-Windows-DHCP-Server"

[WinEventLog://DHCPAdminEvents]
start_from = oldest
disabled = 0

[WinEventLog://Microsoft-Windows-Dhcp-Server/Operational]
start_from = oldest
disabled = 0

[monitor://$WINDIR\System32\DHCP]
disabled = 0
whitelist = DhcpSrvLog*
crcSalt = <SOURCE>
sourcetype = DhcpSrvLog

So I then change my inputs to this to see if I can stop any system events coming whatsoever!

[default]
index = windowsdhcp
_TCP_ROUTING = ssl-production

[monitor://$WINDIR\System32\DHCP]
disabled = 0
whitelist = DhcpSrvLog*
crcSalt = <SOURCE>
sourcetype = DhcpSrvLog

But they keep on coming even though the change is present and btool output does not show any entries for WinEventLog://System

I had to restart the universal forwarder so it to stop picking up all System event log events.  On anoher DHCP server I am back to square one with the original monitoring stanza not working with the whitelisting 

I'm really stumped as to what is going on here! It#s almost like current_only has no effect 😞

[shocko · Accepted solution]

I finally figured it out!  So I suspected that the UF thought it had already collected these events so I had a look into this file %SPLUNK_HOME %\var\lib\splunk\modinputs\WinEventLog\System which basically contains a record where the UF left off on a given WinEventlog i.e. the EventID it last consumed.

<BookmarkList>
  <Bookmark Channel='System' RecordId='1236992' IsCurrent='true'/>
</BookmarkList>

Event 1236992 is one higher than the last instance of the events I wish to whitelist and consume (you can see the EventID in the windows event viewer). I stepped through the following:

1. Stopped the UF
2. Deleted this file
3. Started the UF

I now have all my events!

Note: this may contain duplicates events so we can use delete to remove them based on EventID.  I then deleted other events form my index that I did not require via https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Delete. Be careful if running this!

[rnowitzki]

Hi @shocko,

The first Event is:

<Channel>DhcpAdminEvents</Channel> 

So this one is processed by your DHCPAdminEvents Stanza. There is no white/blacklist, so it is processed.

To allow the second Event, I guess the whitelist entry has to be adjusted. 
I did not use this recently, but looking at the docs it has to be key=regex syntax. 

•  "SourceName" is not the key I guess....isn't it "EventSourceName".  And maybe even something like "System.Provider.EventSourceName"? (not sure about that part right now)
  
  
• In this (2018) post Nick mentions, that quotations in the regex might be an issue. Maybe try to escape them EventSourceName=\"DhcpServer\". Or something like EventSourceName\S\SDhcpServer\S   (just to try if the quotations are an issue also when escaped)