Splunk Search

Universal Forwarder blacklist - Can anyone help with a simple regex?

Path Finder

Hi everyone,

On my Universal Forwarder, I'm able to effectively blacklist Windows event codes when I do it based on the EventCode field. However, when I try to add regex to my blacklist entries it doesn't work.

Essentially, I want to reduce the number of EventCode=4688 entries where the "New Process Name" field is coming from the Splunk client. So let's say, I want to blacklist events where the EventCode=4688 and the New Process Name contains "splunk-winprintmon.exe".

Here's the contents of the actual event:

01/07/2016 12:38:09 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4688
EventType=0
Type=Information
ComputerName=BBLAPTOP.local
TaskCategory=Process Creation
OpCode=Info
RecordNumber=8541755
Keywords=Audit Success
Message=A new process has been created.

Subject:
    Security ID:        NT AUTHORITY\SYSTEM
    Account Name:       BBLAPTOP$
    Account Domain:     LOCAL
    Logon ID:       0x3e7

Process Information:
    New Process ID:     0x6c18
    New Process Name:   C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe
    Token Elevation Type:   TokenElevationTypeDefault (1)
    Creator Process ID: 0x2108
    Process Command Line:   

I've been able to get "matching" regexes created when I try it out on a website like regexpal.com. The regex below matches the event text just fine on their site, but does not work with the Splunk forwarder:

(?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

My inputs.conf contains:

blacklist3 = (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

I've also tried it without the leading (?msi)^ and was not successful. I really need to reduce my licensing volume as I'm frequently in violation, so if anyone has any ideas or solutions I would greatly appreciate it!

1 Solution

Path Finder

All:

Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution:

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)"
blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)"

My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it.

I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!

View solution in original post

Path Finder

So did this end up working for you?

blacklist1 = EventCode="4688" Message="(?s).New Process Name:\s+.(splunk-winprintmon.exe)"

I agree, it is very hard to find clear information on getting the UF to filter even though the capability is there. I would love to know the correct regex to filter out logs with account names that end with $

0 Karma

Path Finder

The way mine works is actually like this:

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon|splunk-MonitorNoHandle)"

So the blacklist feature in the case looks for Windows Security Event Log events with an EventCode of 4688, and then reads into the Message field and uses regex to match the message contents. For this one I believe was blacklisting EC 4688 when the splunk-*mon process was in the message or splunk-MonitorNoHandle was in the message.

Another working example:

blacklist3 = EventCode="4688" Message="(A new process has been created)(?s).*(GoogleUpdate\.exe|FlashPlayerUpdateService\.exe)"

Again, looking at EventCode of 4688, and then checking for a match when the message contains "A new process has been created" and either GoogleUpdate.exe or FlashPlayerUpdateService.exe. If it matches, it gets dropped by the UF.

0 Karma

New Member

We should stop asking folks to filter out at the Indexer when a capability exists at UF. Why have the Indexer incur regex hits and use network bandwidth for nothing (sending over pipe to Indexer that is going to drop on the floor) when a noisy Windows EventID can be filtered at the source?

0 Karma

Contributor

One reason (which has come up on our end) is that the "blacklist" ability for Universal Forwarders is limited to 10 entries (blacklist0-9). We began combining a lot of filters as much as possible, but it was still simpler in some cases just to implement it on the indexer side and be done with it.

0 Karma

Path Finder

All:

Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution:

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)"
blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)"

My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it.

I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!

View solution in original post

Splunk Employee
Splunk Employee

This will do what you're looking for:

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-.+mon\.exe)"
0 Karma

Path Finder

Thanks! I actually got it working using this (right about the same time you posted):

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon)"

Are there any advantages or disadvantages to doing it my way as opposed to your suggestion?

0 Karma

Legend

Okay, I broke down and read all the documentation I could find on filtering Windows inputs. I believe that your syntax is wrong. I am not sure that there is a way to do what you want on the UF.

First, this type of filtering only works on Windows event logs, eg

[WinEventLog://Security]

Second, you need to specify the regular expression against specific fields, which they call "keys", for example:

 blacklist = EventCode=%4688$% User=%lguinn%

I don't see a key for " New Process Name"
Here is an excerpt from inputs.conf.spec

Valid keys for the regex format:
 * The following keys are equivalent to the fields which appear in the text of
  the acquired events: Category CategoryString ComputerName EventCode
  EventType Keywords LogName Message OpCode RecordNumber Sid SidType
  SourceName TaskCategory Type User
 * There are two special keys that do not appear literally in the event.
 * $TimeGenerated : The time that the computer generated the event
 * $Timestamp: The time that the event was received and recorded by the
                Event Log service.
 * EventType is only available on Server 2003 / XP and earlier
 * Type is only available on Server 2008 / Vista and later

Also, I read Monitor Windows event log data

So, unless someone can show that this ever worked in some other way - may I suggest that you do the filtering on the indexer?

0 Karma

Splunk Employee
Splunk Employee

Lisa is totally right, I had forgotten that there are specific keys that we extract for filtering Windows events at the forwarder level. We've got to create RegEx filters for each of those keys.

This should work:

blacklist1 = EventCode="4688" Message="New Process Name:\s+.*(splunk-winprintmon\.exe)"

With this blacklist1 rule, we've got two keys, EventCode and Message. Each of those keys have a RegEx value that is bounded by comma delimiters as required in the spec.

I don't believe that matching for the Message key is anchored, however, if it is, we'll probably need to do something like this to match all of the message text prior to "New Process Name":

blacklist1 = EventCode="4688" Message="(?s).*New Process Name:\s+.*(splunk-winprintmon\.exe)"

However, I'd try the initial RegEx first.

Big thanks to Lisa for pointing out that we've got to work with Keys and not just the raw data!

0 Karma

Path Finder

I've tried posting the solution twice, but it isn't showing up. I don't know if it's stuck awaiting moderation or what....

0 Karma

Path Finder

What was the solution in the end?

0 Karma

SplunkTrust
SplunkTrust

Try using a Transform (on Indexer) to filter the events before indexing (and save on license)

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad#Filter_WMI_and_Even...

0 Karma

Splunk Employee
Splunk Employee

baf879, as a few people have mentioned, you can also use nullQueue routing on the indexer...however, what you're attempting to accomplish is entirely possible via the method you're using. We just need to fix the RegEx matching for it to work. Let's keep the filtering as close to the data as possible...out on the Universal Forwarder.

0 Karma

Legend

Unless there is a very high volume of data to be discarded, I take a different approach. I prefer discarding the data on the indexer, because I (Splunk Admin) own the indexer. It is possible, though unlikely, that someone could muck with the settings on the forwarder, which is on a server that I do not own.

Unless you are discarding at least 50% of the data, there is rarely any performance benefit to discarding the data on the UF.

0 Karma

Path Finder

Thank you for the input. I can see the advantage of "centralizing" the process of discarding events. We use a deployment server so while I may not "own" explicit administrative access to them, I can effect change on them via that method.

Frankly, I'm not sure how much data is being discarded (or the best way to determine that). We had an explosion in our Windows security log events - to the tune of over 100 GB per day. My license is for 50GB so you can see how that is a problem 🙂 . At the indexer, I used props/transforms (with a regex provided from another group at my location) to drop some of our most frequently occurring events. Once that started, my licensing dropped down to just above 50GB per day. Around the same time, I started noticing the indexing latency, so I was trying to see if I could push that filtering capability out to the clients to effectively rule it out as the root cause.

I haven't made changes in production yet, so it remains to be seen if the latency will be affected (if at all). But thanks to everyone's help, I now have a sort of template to follow if and when I need to drop other events.

0 Karma

Contributor

Are you sure that the Universal Forwarder can do this? Unless something has changed, it was my understanding that they do not do the same detailed input parsing as a Heavy Indexer, therefore you wouldn't be able to do a blacklist regex which matches a pattern INSIDE the file. Even the examples in the Docs on Splunk's own site note only matching the filename/path with a regex.

EDIT - Okay, I see (looking at the inputs.conf spec) that this appears to be a specific special case (being able to filter event level data at the UF level) for the [WinEventLog://] input type. Regular [monitor://] types and others still have no ability to blacklist at the event level by regex, correct?

0 Karma

Path Finder

Thanks. I am actually doing this at the indexer currently (for EventCode 4688 where my 3rd party inventory agent is being chatty). My license usage dropped significantly (by half), however, I started seeing some indexing latency. I was trying to eliminate this as a potential cause by pushing it out to the clients. Not sure if it really would cause the latency but figured it was worth investigating.

0 Karma

Splunk Employee
Splunk Employee

Try this:

blacklist3=(?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

The carat (^) you were using between (?msi) and EventCode is causing a matching problem. EventCode does not appear at the beginning of your event, its the 5th line.

Also, make sure your blacklists are sequential. For example, you should go:

blacklist1
blacklist2
blacklist3

not

blacklist1
blacklist3
blacklist4

0 Karma

Path Finder

OK great. Let me try removing that carat. My blacklist items are sequential (thanks for the reminder) - this one in particular was #3 on my list. The two preceding it were for Active Directory (4662 and 566); my Splunk PS engineer put that stanza there when she set up our Splunk cluster.

The blacklist section actually looks like this:

blacklist = 4656,5145,4985,4904,4905,4945,4957,5033,5024,5058,5440,5441,5442,5444,5632,6281,5031,5145
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
0 Karma

Splunk Employee
Splunk Employee

Also, the values above for blacklist1 and blacklist2 should match

0 Karma