topic Re: Universal Forwarder blacklist - Can anyone help with a simple regex? in Splunk Search

Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 17:57:29 GMT

Hi everyone,

On my Universal Forwarder, I'm able to effectively blacklist Windows event codes when I do it based on the EventCode field. However, when I try to add regex to my blacklist entries it doesn't work.

Essentially, I want to reduce the number of EventCode=4688 entries where the "New Process Name" field is coming from the Splunk client. So let's say, I want to blacklist events where the EventCode=4688 and the New Process Name contains "splunk-winprintmon.exe".

Here's the contents of the actual event:

01/07/2016 12:38:09 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4688
EventType=0
Type=Information
ComputerName=BBLAPTOP.local
TaskCategory=Process Creation
OpCode=Info
RecordNumber=8541755
Keywords=Audit Success
Message=A new process has been created.

Subject:
    Security ID:        NT AUTHORITY\SYSTEM
    Account Name:       BBLAPTOP$
    Account Domain:     LOCAL
    Logon ID:       0x3e7

Process Information:
    New Process ID:     0x6c18
    New Process Name:   C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe
    Token Elevation Type:   TokenElevationTypeDefault (1)
    Creator Process ID: 0x2108
    Process Command Line:

I've been able to get "matching" regexes created when I try it out on a website like regexpal.com. The regex below matches the event text just fine on their site, but does not work with the Splunk forwarder:

(?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

My inputs.conf contains:

blacklist3 = (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

I've also tried it without the leading (?msi)^ and was not successful. I really need to reduce my licensing volume as I'm frequently in violation, so if anyone has any ideas or solutions I would greatly appreciate it!

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

lguinn2 — Thu, 07 Jan 2016 19:31:34 GMT

If you are having trouble with this on the forwarder, why not do this on the indexer instead? The indexer is capable of handling more advanced criteria - and it will not affect your license. The following example will eliminate the events before they are indexed:

props.conf

[sourcetypeNameofEventLog]
TRANSFORMS-eliminate = remove_winprintmon

transforms.conf

[remove_winprintmon]
SOURCE=_raw
REGEX= (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe)
DEST=queue
FORMAT=nullQueue

This will pull matching events from the index queue and trash them. Note that the transform will only run against the inbound data that matches the sourcetype. BTW, regexes in transforms.conf are unanchored by default, so you don't need the .* at the end...

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Thu, 07 Jan 2016 19:32:58 GMT

Try this:

blacklist3=(?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).*

The carat (^) you were using between (?msi) and EventCode is causing a matching problem. EventCode does not appear at the beginning of your event, its the 5th line.

Also, make sure your blacklists are sequential. For example, you should go:

blacklist1
blacklist2
blacklist3

not

blacklist1
blacklist3
blacklist4

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

somesoni2 — Thu, 07 Jan 2016 19:42:20 GMT

Try using a Transform (on Indexer) to filter the events before indexing (and save on license)

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad#Filter_WMI_and_Event_Log_events

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 19:47:03 GMT

Thanks. I am actually doing this at the indexer currently (for EventCode 4688 where my 3rd party inventory agent is being chatty). My license usage dropped significantly (by half), however, I started seeing some indexing latency. I was trying to eliminate this as a potential cause by pushing it out to the clients. Not sure if it really would cause the latency but figured it was worth investigating.

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Thu, 07 Jan 2016 19:47:05 GMT

baf879, as a few people have mentioned, you can also use nullQueue routing on the indexer...however, what you're attempting to accomplish is entirely possible via the method you're using. We just need to fix the RegEx matching for it to work. Let's keep the filtering as close to the data as possible...out on the Universal Forwarder.

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 19:50:23 GMT

Thank you for the tip regarding the .* at the end, I'll go in and remove that. There's one more thing I learned about regex today !

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 19:54:39 GMT

OK great. Let me try removing that carat. My blacklist items are sequential (thanks for the reminder) - this one in particular was #3 on my list. The two preceding it were for Active Directory (4662 and 566); my Splunk PS engineer put that stanza there when she set up our Splunk cluster.

The blacklist section actually looks like this:

blacklist = 4656,5145,4985,4904,4905,4945,4957,5033,5024,5058,5440,5441,5442,5444,5632,6281,5031,5145
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Thu, 07 Jan 2016 19:59:30 GMT

Ok, that looks good...except that I doubt the RegEx in blacklist1 and blacklist2 will match anything

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 20:15:24 GMT

It looks like your blacklist3 suggestion might have gotten messed up, as it's missing some asterisks that I had. Unless you purposely left them out...so should blacklist3 actually look like this:

blacklist3 = (?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe)

This would omit the first carat, as well as the "trailing" .* . I also put the backslash between splunk-winprintmon and the .exe .

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

lguinn2 — Thu, 07 Jan 2016 20:30:25 GMT

To investigate the latency, you might turn on the Distributed Management Console on your indexer (if you haven't already) and look at the indexing dashboards, particular the queues...

For more information, go to http://conf.splunk.com/speakers.html# and search the page for the topic "How splunkd Works"
It's a great presentation, if you haven't seen it before. There is both a recording and a slide download link.

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Tue, 29 Sep 2020 08:19:43 GMT

Yes, you're right. Somehow the formatting got messed up. The line should look like the one you've posted.

blacklist3 = (?msi)EventCode=(4688).^.*New Process Name:\s+.(splunk-winprintmon.exe)

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Thu, 07 Jan 2016 21:02:01 GMT

No luck this time. I did comment out the entries you said probably aren't working, and bumped this up to #1. So my inputs.conf contains this line:

blacklist1 = (?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon.exe)

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

tmeader — Thu, 07 Jan 2016 22:33:30 GMT

Are you sure that the Universal Forwarder can do this? Unless something has changed, it was my understanding that they do not do the same detailed input parsing as a Heavy Indexer, therefore you wouldn't be able to do a blacklist regex which matches a pattern INSIDE the file. Even the examples in the Docs on Splunk's own site note only matching the filename/path with a regex.

EDIT - Okay, I see (looking at the inputs.conf spec) that this appears to be a specific special case (being able to filter event level data at the UF level) for the [WinEventLog://] input type. Regular [monitor://] types and others still have no ability to blacklist at the event level by regex, correct?

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

lguinn2 — Fri, 08 Jan 2016 01:21:41 GMT

Unless there is a very high volume of data to be discarded, I take a different approach. I prefer discarding the data on the indexer, because I (Splunk Admin) own the indexer. It is possible, though unlikely, that someone could muck with the settings on the forwarder, which is on a server that I do not own.

Unless you are discarding at least 50% of the data, there is rarely any performance benefit to discarding the data on the UF.

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

lguinn2 — Fri, 08 Jan 2016 01:39:26 GMT

Okay, I broke down and read all the documentation I could find on filtering Windows inputs. I believe that your syntax is wrong. I am not sure that there is a way to do what you want on the UF.

First, this type of filtering only works on Windows event logs, eg

[WinEventLog://Security]

Second, you need to specify the regular expression against specific fields, which they call "keys", for example:

 blacklist = EventCode=%4688$% User=%lguinn%

I don't see a key for " New Process Name"
Here is an excerpt from inputs.conf.spec

Valid keys for the regex format:
 * The following keys are equivalent to the fields which appear in the text of
  the acquired events: Category CategoryString ComputerName EventCode
  EventType Keywords LogName Message OpCode RecordNumber Sid SidType
  SourceName TaskCategory Type User
 * There are two special keys that do not appear literally in the event.
 * $TimeGenerated : The time that the computer generated the event
 * $Timestamp: The time that the event was received and recorded by the
                Event Log service.
 * EventType is only available on Server 2003 / XP and earlier
 * Type is only available on Server 2008 / Vista and later

Also, I read Monitor Windows event log data

So, unless someone can show that this ever worked in some other way - may I suggest that you do the filtering on the indexer?

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Fri, 08 Jan 2016 14:46:48 GMT

Lisa is totally right, I had forgotten that there are specific keys that we extract for filtering Windows events at the forwarder level. We've got to create RegEx filters for each of those keys.

This should work:

blacklist1 = EventCode="4688" Message="New Process Name:\s+.*(splunk-winprintmon\.exe)"

With this blacklist1 rule, we've got two keys, EventCode and Message. Each of those keys have a RegEx value that is bounded by comma delimiters as required in the spec.

I don't believe that matching for the Message key is anchored, however, if it is, we'll probably need to do something like this to match all of the message text prior to "New Process Name":

blacklist1 = EventCode="4688" Message="(?s).*New Process Name:\s+.*(splunk-winprintmon\.exe)"

However, I'd try the initial RegEx first.

Big thanks to Lisa for pointing out that we've got to work with Keys and not just the raw data!

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Fri, 08 Jan 2016 14:48:01 GMT

See my comments below under Lisa's answer. I made a mistake above and we actually need to work with a specific set of Keys here, not just the raw event.

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

jchampagne_splu — Fri, 08 Jan 2016 14:52:24 GMT

Also, the values above for blacklist1 and blacklist2 should match

Re: Universal Forwarder blacklist - Can anyone help with a simple regex?

baf879 — Fri, 08 Jan 2016 14:59:20 GMT

All:

Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution:

blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)"
blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)"

My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it.

I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!