Getting Data In

Redirect to Null Queue is not working

bailmon
Explorer

I'm trying to redirect all 5145 events (from WinEventLog:Security) and all Security events from 'SYSTEM' (or another account called digitalsender). I've tried several variations of the entries below, but nothing seems to work.

Here is the relevant parts of my configs..

props.conf

[source::WinEventLog:Security]
TRANSFORMS-null = null_excluded_users, null_excluded_events

transforms.conf

[null_excluded_users]
case_sensitive_match = false
REGEX = (?m)LogName=Security[^$]+(Security ID:(.*)(AUTHORITY\\SYSTEM|digitalsender)|User=(SYSTEM|digitalsender))
DEST_KEY = queue
FORMAT = nullQueue


[null_excluded_events] 
case_sensitive_match = false
REGEX = (?m)EventCode=(5145||)
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
1 Solution

kristian_kolb
Ultra Champion

Hi,

A few things;

It may not have bearing here, but the case_sensitive_match has nothing to do with regex filtering to queues. It is to be used when looking up stuff from a lookup-table.
Set (?mi) in the beginning of your regex to achieve the case insensitivity matching.

Isn't LogName=Security part of the regex a bit redundant? The transform is called from a props.conf setting based off the sourcetype, which is WinEventLog:Security, so wouldn't LogName always be 'Security'?

I think the regex would work better if you split it two separate nullQueue transforms, along the lines of;

[null_seqid]
REGEX = (?m)Security ID:\s+(NT AUTHORITY\\SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

[null_user]
REGEX = (?m)User=(SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

Also, you can remove the two pipes (and the parentheses) in the regex section of [null_excluded_events].


UPDATE:

Ok, that might explain a lot, depending on which 'type' of splunk you installed on the machine generating the data. (Universal Forwarder, or a regular Splunk installation)

The reason for that is that incoming data passes through a couple of phases; Input, Parsing, Indexing and Search. These type of nullQueue routing operations take place during the Parsing phase, and ... the first parsing-capable component in the chain between source file and index on disk will perform that parsing. Once data has passed through the parsing phase, it will be marked as such, and will not be parsed again (unless you explicitly instruct it to do so, but that is not a recommended practice)

So if you installed the Universal Forwarder, and told it to get the data and forward it the Splunk Indexer, your configuration should go in the Indexers props/transforms.conf files. (Universal Forwarder cannot do parsing, and will just ignore any such settings).

If you installed a full Splunk, you should set these type of configurations on the sending side, since a full Splunk installtion can and will do the parsing. A full Splunk installtion configured to forward events rather than index them itself is commonly called a Heavy Forwarder.

You should perhaps read the following, which might make it more clear;

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Hope this helps,

Kristian

View solution in original post

kristian_kolb
Ultra Champion

Hi,

A few things;

It may not have bearing here, but the case_sensitive_match has nothing to do with regex filtering to queues. It is to be used when looking up stuff from a lookup-table.
Set (?mi) in the beginning of your regex to achieve the case insensitivity matching.

Isn't LogName=Security part of the regex a bit redundant? The transform is called from a props.conf setting based off the sourcetype, which is WinEventLog:Security, so wouldn't LogName always be 'Security'?

I think the regex would work better if you split it two separate nullQueue transforms, along the lines of;

[null_seqid]
REGEX = (?m)Security ID:\s+(NT AUTHORITY\\SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

[null_user]
REGEX = (?m)User=(SYSTEM|digitalsender) 
DEST_KEY = queue
FORMAT = nullQueue

Also, you can remove the two pipes (and the parentheses) in the regex section of [null_excluded_events].


UPDATE:

Ok, that might explain a lot, depending on which 'type' of splunk you installed on the machine generating the data. (Universal Forwarder, or a regular Splunk installation)

The reason for that is that incoming data passes through a couple of phases; Input, Parsing, Indexing and Search. These type of nullQueue routing operations take place during the Parsing phase, and ... the first parsing-capable component in the chain between source file and index on disk will perform that parsing. Once data has passed through the parsing phase, it will be marked as such, and will not be parsed again (unless you explicitly instruct it to do so, but that is not a recommended practice)

So if you installed the Universal Forwarder, and told it to get the data and forward it the Splunk Indexer, your configuration should go in the Indexers props/transforms.conf files. (Universal Forwarder cannot do parsing, and will just ignore any such settings).

If you installed a full Splunk, you should set these type of configurations on the sending side, since a full Splunk installtion can and will do the parsing. A full Splunk installtion configured to forward events rather than index them itself is commonly called a Heavy Forwarder.

You should perhaps read the following, which might make it more clear;

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Hope this helps,

Kristian

bailmon
Explorer

Hi Kristian,

Sorry it took me so long to get back with you. 1.) I had to take in all of the information, and 2.) today was a crazy day.

But Eureka... I think you've solved the problem. I did have it installed as a 'full install' on that server. I changed it to a 'light forwarder' and viola. The events have stopped coming.

Also. The link you sent is like the Rosetta Stone for understanding the data life cycle.

Thank You So much..

0 Karma

kristian_kolb
Ultra Champion

see update above.

0 Karma

bailmon
Explorer

Thank you Kristian,

I'll read up on it. Just to clarify how the events are coming. I installed Splunk on the server that generates those events and set it to forward all of it's events to the collector (port 9997)

0 Karma

kristian_kolb
Ultra Champion

Hm, the REGEX for filtering out the 5145 messages should be fairly straightforward.

This could not be a WMI-issue? The source would be different when collected remotely, compared to being collected through a forwarder. At least it used to be like that, WMI vs. WinEventLog:Security.

You could always check out:

http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/MonitorWMIdata#Fields_for_WMI_data

Having the wrong source/sourcetype would cause the props/transform NOT to be triggered.

/K

0 Karma

bailmon
Explorer

Hi Kristian,

Thank you. It's well thought out, but unfortunately, the events are still coming in. I'm monitoring 30 seconds realtime and watching the 5145 events just pour in. I copied the [null_user] and [null_seqid] directly into the transforms.conf and added the stanza in the props.conf.

You are right about the 'LogName=Security' being redundant.

Thanks for the information about the 'case_sensitive_match' (I did not know that).

I wish I understood the process (route these events take to get into Splunk) better.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...