Hi everyone,
On my Universal Forwarder, I'm able to effectively blacklist Windows event codes when I do it based on the EventCode field. However, when I try to add regex to my blacklist entries it doesn't work.
Essentially, I want to reduce the number of EventCode=4688 entries where the "New Process Name" field is coming from the Splunk client. So let's say, I want to blacklist events where the EventCode=4688 and the New Process Name contains "splunk-winprintmon.exe".
Here's the contents of the actual event:
01/07/2016 12:38:09 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4688
EventType=0
Type=Information
ComputerName=BBLAPTOP.local
TaskCategory=Process Creation
OpCode=Info
RecordNumber=8541755
Keywords=Audit Success
Message=A new process has been created.
Subject:
Security ID: NT AUTHORITY\SYSTEM
Account Name: BBLAPTOP$
Account Domain: LOCAL
Logon ID: 0x3e7
Process Information:
New Process ID: 0x6c18
New Process Name: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe
Token Elevation Type: TokenElevationTypeDefault (1)
Creator Process ID: 0x2108
Process Command Line:
I've been able to get "matching" regexes created when I try it out on a website like regexpal.com. The regex below matches the event text just fine on their site, but does not work with the Splunk forwarder:
(?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*
My inputs.conf contains:
blacklist3 = (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*
I've also tried it without the leading (?msi)^ and was not successful. I really need to reduce my licensing volume as I'm frequently in violation, so if anyone has any ideas or solutions I would greatly appreciate it!
All:
Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution:
blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)"
blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)"
My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it.
I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!
So did this end up working for you?
blacklist1 = EventCode="4688" Message="(?s).New Process Name:\s+.(splunk-winprintmon.exe)"
I agree, it is very hard to find clear information on getting the UF to filter even though the capability is there. I would love to know the correct regex to filter out logs with account names that end with $
The way mine works is actually like this:
blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon|splunk-MonitorNoHandle)"
So the blacklist feature in the case looks for Windows Security Event Log events with an EventCode of 4688, and then reads into the Message field and uses regex to match the message contents. For this one I believe was blacklisting EC 4688 when the splunk-*mon process was in the message or splunk-MonitorNoHandle was in the message.
Another working example:
blacklist3 = EventCode="4688" Message="(A new process has been created)(?s).*(GoogleUpdate\.exe|FlashPlayerUpdateService\.exe)"
Again, looking at EventCode of 4688, and then checking for a match when the message contains "A new process has been created" and either GoogleUpdate.exe or FlashPlayerUpdateService.exe. If it matches, it gets dropped by the UF.
We should stop asking folks to filter out at the Indexer when a capability exists at UF. Why have the Indexer incur regex hits and use network bandwidth for nothing (sending over pipe to Indexer that is going to drop on the floor) when a noisy Windows EventID can be filtered at the source?
One reason (which has come up on our end) is that the "blacklist" ability for Universal Forwarders is limited to 10 entries (blacklist0-9). We began combining a lot of filters as much as possible, but it was still simpler in some cases just to implement it on the indexer side and be done with it.
All:
Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution:
blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)"
blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)"
My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it.
I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!
This will do what you're looking for:
blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-.+mon\.exe)"
Thanks! I actually got it working using this (right about the same time you posted):
blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon)"
Are there any advantages or disadvantages to doing it my way as opposed to your suggestion?
Okay, I broke down and read all the documentation I could find on filtering Windows inputs. I believe that your syntax is wrong. I am not sure that there is a way to do what you want on the UF.
First, this type of filtering only works on Windows event logs, eg
[WinEventLog://Security]
Second, you need to specify the regular expression against specific fields, which they call "keys", for example:
blacklist = EventCode=%4688$% User=%lguinn%
I don't see a key for " New Process Name"
Here is an excerpt from inputs.conf.spec
Valid keys for the regex format:
* The following keys are equivalent to the fields which appear in the text of
the acquired events: Category CategoryString ComputerName EventCode
EventType Keywords LogName Message OpCode RecordNumber Sid SidType
SourceName TaskCategory Type User
* There are two special keys that do not appear literally in the event.
* $TimeGenerated : The time that the computer generated the event
* $Timestamp: The time that the event was received and recorded by the
Event Log service.
* EventType is only available on Server 2003 / XP and earlier
* Type is only available on Server 2008 / Vista and later
Also, I read Monitor Windows event log data
So, unless someone can show that this ever worked in some other way - may I suggest that you do the filtering on the indexer?
Lisa is totally right, I had forgotten that there are specific keys that we extract for filtering Windows events at the forwarder level. We've got to create RegEx filters for each of those keys.
This should work:
blacklist1 = EventCode="4688" Message="New Process Name:\s+.*(splunk-winprintmon\.exe)"
With this blacklist1 rule, we've got two keys, EventCode and Message. Each of those keys have a RegEx value that is bounded by comma delimiters as required in the spec.
I don't believe that matching for the Message key is anchored, however, if it is, we'll probably need to do something like this to match all of the message text prior to "New Process Name":
blacklist1 = EventCode="4688" Message="(?s).*New Process Name:\s+.*(splunk-winprintmon\.exe)"
However, I'd try the initial RegEx first.
Big thanks to Lisa for pointing out that we've got to work with Keys and not just the raw data!
I've tried posting the solution twice, but it isn't showing up. I don't know if it's stuck awaiting moderation or what....
What was the solution in the end?
Try using a Transform (on Indexer) to filter the events before indexing (and save on license)
baf879, as a few people have mentioned, you can also use nullQueue routing on the indexer...however, what you're attempting to accomplish is entirely possible via the method you're using. We just need to fix the RegEx matching for it to work. Let's keep the filtering as close to the data as possible...out on the Universal Forwarder.
Unless there is a very high volume of data to be discarded, I take a different approach. I prefer discarding the data on the indexer, because I (Splunk Admin) own the indexer. It is possible, though unlikely, that someone could muck with the settings on the forwarder, which is on a server that I do not own.
Unless you are discarding at least 50% of the data, there is rarely any performance benefit to discarding the data on the UF.
Thank you for the input. I can see the advantage of "centralizing" the process of discarding events. We use a deployment server so while I may not "own" explicit administrative access to them, I can effect change on them via that method.
Frankly, I'm not sure how much data is being discarded (or the best way to determine that). We had an explosion in our Windows security log events - to the tune of over 100 GB per day. My license is for 50GB so you can see how that is a problem 🙂 . At the indexer, I used props/transforms (with a regex provided from another group at my location) to drop some of our most frequently occurring events. Once that started, my licensing dropped down to just above 50GB per day. Around the same time, I started noticing the indexing latency, so I was trying to see if I could push that filtering capability out to the clients to effectively rule it out as the root cause.
I haven't made changes in production yet, so it remains to be seen if the latency will be affected (if at all). But thanks to everyone's help, I now have a sort of template to follow if and when I need to drop other events.
Are you sure that the Universal Forwarder can do this? Unless something has changed, it was my understanding that they do not do the same detailed input parsing as a Heavy Indexer, therefore you wouldn't be able to do a blacklist regex which matches a pattern INSIDE the file. Even the examples in the Docs on Splunk's own site note only matching the filename/path with a regex.
EDIT - Okay, I see (looking at the inputs.conf spec) that this appears to be a specific special case (being able to filter event level data at the UF level) for the [WinEventLog://] input type. Regular [monitor://] types and others still have no ability to blacklist at the event level by regex, correct?
Thanks. I am actually doing this at the indexer currently (for EventCode 4688 where my 3rd party inventory agent is being chatty). My license usage dropped significantly (by half), however, I started seeing some indexing latency. I was trying to eliminate this as a potential cause by pushing it out to the clients. Not sure if it really would cause the latency but figured it was worth investigating.
Try this:
blacklist3=(?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*
The carat (^) you were using between (?msi) and EventCode is causing a matching problem. EventCode does not appear at the beginning of your event, its the 5th line.
Also, make sure your blacklists are sequential. For example, you should go:
blacklist1
blacklist2
blacklist3
not
blacklist1
blacklist3
blacklist4
OK great. Let me try removing that carat. My blacklist items are sequential (thanks for the reminder) - this one in particular was #3 on my list. The two preceding it were for Active Directory (4662 and 566); my Splunk PS engineer put that stanza there when she set up our Splunk cluster.
The blacklist section actually looks like this:
blacklist = 4656,5145,4985,4904,4905,4945,4957,5033,5024,5058,5440,5441,5442,5444,5632,6281,5031,5145
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
Also, the values above for blacklist1 and blacklist2 should match