Getting Data In

How do I configure Splunk to filter out events I don’t want to index?

Communicator

A lot of the Windows Security Events we see in Splunk, come from system-users that we're not interested in. I know there's a way to configure Splunk to filter out events based on the event content, but I can't get it to work.

In the example event below, I dont want to index events produced bt the

04/27/10 03:15:38 PM  
LogName=Security  
SourceName=Microsoft Windows security auditing.  
EventCode=4634  
EventType=0  
Type=Information  
ComputerName=my.testmachine.env  
TaskCategory=Logoff  
OpCode=Info  
RecordNumber=12345678  
Keywords=Audit Success  
Message=An account was logged off.

Subject:  
Security ID:            L-O-O-K-Y  
Account Name:           MAN-PAGE-H3LP$  
Account Domain:         TEST  
Logon ID:               0x123456789  
Logon Type:                     7  

My current configuration is -

inputs.conf:  
[WinEventLog:Security]  
disabled = 0  

props.conf:  
[source::WinEventLog:Security]  
TRANSFORMS-Event=MachineNamePurge  
blacklist = Account\sName:\s+\w+\$ 
Tags (3)
1 Solution

Splunk Employee
Splunk Employee

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved.

This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here.

In the config below, we have your regular inputs.conf entry -

Inputs.conf 
[WinEventLog:Security] 
disabled = 0 

As you can see, nothing relating to blacklisting here. Next comes the props.conf entry -

Props.conf 
[source::WinEventLog:Security] 
TRANSFORMS-nullQ= nullFilter 

and then the transforms.conf entry is where the actual REGEX and actions are defined -

Transforms.conf 
[nullFilter] 
REGEX = Account\sName:[\s|\w|-]+\$ 
DEST_KEY=queue 
FORMAT = nullQueue 

So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS.

In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal.

All of the configuration steps around this is documented here

View solution in original post

Path Finder

While I have not been able to test this myself, the Splunk Cheatsheet (latest ver 2.21, Feb 2019) from MalwareArchaeology.com includes an example of how to do this in Splunk with some snippets here. Note that only 10 blacklists can be entered per

[WinEventLog://Security]
disabled=0
current_only=1
blacklist = 4689,5158
blacklist1 = EventCode="4688" Message="(?:New Process
Name:).+(?:SplunkUniversalForwarder\\bin\\splunk.exe)|.+(?:SplunkUniversalForwarder\\bin\\splunkd.exe)|.+(?:Splunk
UniversalForwarder\\bin\\btool.exe)"
...
blacklist4 = EventCode="4688" Message="(?:Process Command Line:).+(?:system32\\SearchFilterHost.exe)|.+(?:find
/i)|.+(?:Google\\Update\\GoogleUpdate.exe)|.+(?:WINDOWS\\system32\\conhost.exe)"

From the splunk docs on filtering Windows Events with inputs.conf (keywords: whitelist, blacklist), it notes:

You can have multiple key/regular expression sets in a single advanced filtering entry. Splunk software conjuncts the sets logically. This means that the entry is valid only if all of the sets in the entry are true.
You can specify up to 10 blacklists per stanza by adding a number to the end of the blacklist attribute, for example blacklist1...blacklist9.

0 Karma

Communicator

Is it possible to blacklist an event from one index and whitelist same event for another index?

0 Karma

Splunk Employee
Splunk Employee

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved.

This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here.

In the config below, we have your regular inputs.conf entry -

Inputs.conf 
[WinEventLog:Security] 
disabled = 0 

As you can see, nothing relating to blacklisting here. Next comes the props.conf entry -

Props.conf 
[source::WinEventLog:Security] 
TRANSFORMS-nullQ= nullFilter 

and then the transforms.conf entry is where the actual REGEX and actions are defined -

Transforms.conf 
[nullFilter] 
REGEX = Account\sName:[\s|\w|-]+\$ 
DEST_KEY=queue 
FORMAT = nullQueue 

So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS.

In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal.

All of the configuration steps around this is documented here

View solution in original post