Getting Data In

How do I configure Splunk to filter out events I don’t want to index?

mctester
Communicator

A lot of the Windows Security Events we see in Splunk, come from system-users that we're not interested in. I know there's a way to configure Splunk to filter out events based on the event content, but I can't get it to work.

In the example event below, I dont want to index events produced bt the

04/27/10 03:15:38 PM  
LogName=Security  
SourceName=Microsoft Windows security auditing.  
EventCode=4634  
EventType=0  
Type=Information  
ComputerName=my.testmachine.env  
TaskCategory=Logoff  
OpCode=Info  
RecordNumber=12345678  
Keywords=Audit Success  
Message=An account was logged off.

Subject:  
Security ID:            L-O-O-K-Y  
Account Name:           MAN-PAGE-H3LP$  
Account Domain:         TEST  
Logon ID:               0x123456789  
Logon Type:                     7  

My current configuration is -

inputs.conf:  
[WinEventLog:Security]  
disabled = 0  

props.conf:  
[source::WinEventLog:Security]  
TRANSFORMS-Event=MachineNamePurge  
blacklist = Account\sName:\s+\w+\$ 
Tags (3)
1 Solution

Mick
Splunk Employee
Splunk Employee

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved.

This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here.

In the config below, we have your regular inputs.conf entry -

Inputs.conf 
[WinEventLog:Security] 
disabled = 0 

As you can see, nothing relating to blacklisting here. Next comes the props.conf entry -

Props.conf 
[source::WinEventLog:Security] 
TRANSFORMS-nullQ= nullFilter 

and then the transforms.conf entry is where the actual REGEX and actions are defined -

Transforms.conf 
[nullFilter] 
REGEX = Account\sName:[\s|\w|-]+\$ 
DEST_KEY=queue 
FORMAT = nullQueue 

So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS.

In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal.

All of the configuration steps around this is documented here

View solution in original post

bwlm
Path Finder

While I have not been able to test this myself, the Splunk Cheatsheet (latest ver 2.21, Feb 2019) from MalwareArchaeology.com includes an example of how to do this in Splunk with some snippets here. Note that only 10 blacklists can be entered per

[WinEventLog://Security]
disabled=0
current_only=1
blacklist = 4689,5158
blacklist1 = EventCode="4688" Message="(?:New Process
Name:).+(?:SplunkUniversalForwarder\\bin\\splunk.exe)|.+(?:SplunkUniversalForwarder\\bin\\splunkd.exe)|.+(?:Splunk
UniversalForwarder\\bin\\btool.exe)"
...
blacklist4 = EventCode="4688" Message="(?:Process Command Line:).+(?:system32\\SearchFilterHost.exe)|.+(?:find
/i)|.+(?:Google\\Update\\GoogleUpdate.exe)|.+(?:WINDOWS\\system32\\conhost.exe)"

From the splunk docs on filtering Windows Events with inputs.conf (keywords: whitelist, blacklist), it notes:

You can have multiple key/regular expression sets in a single advanced filtering entry. Splunk software conjuncts the sets logically. This means that the entry is valid only if all of the sets in the entry are true.
You can specify up to 10 blacklists per stanza by adding a number to the end of the blacklist attribute, for example blacklist1...blacklist9.

0 Karma

ss026381
Communicator

Is it possible to blacklist an event from one index and whitelist same event for another index?

0 Karma

Mick
Splunk Employee
Splunk Employee

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved.

This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here.

In the config below, we have your regular inputs.conf entry -

Inputs.conf 
[WinEventLog:Security] 
disabled = 0 

As you can see, nothing relating to blacklisting here. Next comes the props.conf entry -

Props.conf 
[source::WinEventLog:Security] 
TRANSFORMS-nullQ= nullFilter 

and then the transforms.conf entry is where the actual REGEX and actions are defined -

Transforms.conf 
[nullFilter] 
REGEX = Account\sName:[\s|\w|-]+\$ 
DEST_KEY=queue 
FORMAT = nullQueue 

So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS.

In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal.

All of the configuration steps around this is documented here

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...