Getting Data In
Highlighted

Redirect to Null Queue is not working

Explorer

I'm trying to redirect all 5145 events (from WinEventLog:Security) and all Security events from 'SYSTEM' (or another account called digitalsender). I've tried several variations of the entries below, but nothing seems to work.

Here is the relevant parts of my configs..

props.conf

[source::WinEventLog:Security]
TRANSFORMS-null = null_excluded_users, null_excluded_events

transforms.conf

[null_excluded_users]
case_sensitive_match = false
REGEX = (?m)LogName=Security[^$]+(Security ID:(.*)(AUTHORITY\\SYSTEM|digitalsender)|User=(SYSTEM|digitalsender))
DEST_KEY = queue
FORMAT = nullQueue


[null_excluded_events] 
case_sensitive_match = false
REGEX = (?m)EventCode=(5145||)
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

Re: Redirect to Null Queue is not working

Ultra Champion

Hi,

A few things;

It may not have bearing here, but the case_sensitive_match has nothing to do with regex filtering to queues. It is to be used when looking up stuff from a lookup-table.
Set (?mi) in the beginning of your regex to achieve the case insensitivity matching.

Isn't LogName=Security part of the regex a bit redundant? The transform is called from a props.conf setting based off the sourcetype, which is WinEventLog:Security, so wouldn't LogName always be 'Security'?

I think the regex would work better if you split it two separate nullQueue transforms, along the lines of;

[null_seqid]
REGEX = (?m)Security ID:\s+(NT AUTHORITY\\SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

[null_user]
REGEX = (?m)User=(SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

Also, you can remove the two pipes (and the parentheses) in the regex section of [null_excluded_events].


UPDATE:

Ok, that might explain a lot, depending on which 'type' of splunk you installed on the machine generating the data. (Universal Forwarder, or a regular Splunk installation)

The reason for that is that incoming data passes through a couple of phases; Input, Parsing, Indexing and Search. These type of nullQueue routing operations take place during the Parsing phase, and ... the first parsing-capable component in the chain between source file and index on disk will perform that parsing. Once data has passed through the parsing phase, it will be marked as such, and will not be parsed again (unless you explicitly instruct it to do so, but that is not a recommended practice)

So if you installed the Universal Forwarder, and told it to get the data and forward it the Splunk Indexer, your configuration should go in the Indexers props/transforms.conf files. (Universal Forwarder cannot do parsing, and will just ignore any such settings).

If you installed a full Splunk, you should set these type of configurations on the sending side, since a full Splunk installtion can and will do the parsing. A full Splunk installtion configured to forward events rather than index them itself is commonly called a Heavy Forwarder.

You should perhaps read the following, which might make it more clear;

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Hope this helps,

Kristian

View solution in original post

Highlighted

Re: Redirect to Null Queue is not working

Explorer

Hi Kristian,

Thank you. It's well thought out, but unfortunately, the events are still coming in. I'm monitoring 30 seconds realtime and watching the 5145 events just pour in. I copied the [nulluser] and [nullseqid] directly into the transforms.conf and added the stanza in the props.conf.

You are right about the 'LogName=Security' being redundant.

Thanks for the information about the 'casesensitivematch' (I did not know that).

I wish I understood the process (route these events take to get into Splunk) better.

0 Karma
Highlighted

Re: Redirect to Null Queue is not working

Ultra Champion

Hm, the REGEX for filtering out the 5145 messages should be fairly straightforward.

This could not be a WMI-issue? The source would be different when collected remotely, compared to being collected through a forwarder. At least it used to be like that, WMI vs. WinEventLog:Security.

You could always check out:

http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/MonitorWMIdata#Fields_for_WMI_data

Having the wrong source/sourcetype would cause the props/transform NOT to be triggered.

/K

0 Karma
Highlighted

Re: Redirect to Null Queue is not working

Explorer

Thank you Kristian,

I'll read up on it. Just to clarify how the events are coming. I installed Splunk on the server that generates those events and set it to forward all of it's events to the collector (port 9997)

0 Karma
Highlighted

Re: Redirect to Null Queue is not working

Ultra Champion

see update above.

0 Karma
Highlighted

Re: Redirect to Null Queue is not working

Explorer

Hi Kristian,

Sorry it took me so long to get back with you. 1.) I had to take in all of the information, and 2.) today was a crazy day.

But Eureka... I think you've solved the problem. I did have it installed as a 'full install' on that server. I changed it to a 'light forwarder' and viola. The events have stopped coming.

Also. The link you sent is like the Rosetta Stone for understanding the data life cycle.

Thank You So much..

0 Karma