Getting Data In

Reqex filter in transforms.conf not working

Explorer

At the indexer, I am trying to exclude event records from incoming windows logs that have Logon Type=3. Below is the configuration that I have, but doesn't seem to work.

$SPLUNK_HOME/etc/system/local/props.conf

[WinEventLog:Security]    
TRANSFORMS-security= events-null, events-null3, events-filter

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-null3]
REGEX=Logon Type=\s*(3)\D
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue

The [events-filter] stanza works. I only see the events listed. The [events-null3] doesn't work. I've tried a variety of regex variations. Many came from different questions posted on this forum. Here are the ones I've tried:

REGEX=Logon_Type=\s*(3)\D
REGEX=Logon Type=\s*(3)\D
REGEX=Logon_Type=\s*(3)
REGEX=(?msi).Logon\sType:\s3D
REGEX=(?msi).Logon\sType:\s3
REGEX=(?m).*Logon\sType:\s+3.*
REGEX=(?m).Logon\sType:\s+3
REGEX = (?msi).*Logon Type:\s*(2|10)

Below is a sample of an event I am trying to filter out.


12/02/2013 08:45:43 AM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4634
EventType=0
Type=Information
ComputerName=P-UMS03.p-umhs.med.umich.edu
TaskCategory=Logoff
OpCode=Info
RecordNumber=5744631540
Keywords=Audit Success
Message=An account was logged off.

Subject:
Security ID:        P-UMS\USPCNTBSA1$
Account Name:       USPCNTBSA1$
Account Domain:     P-UMS
Logon ID:       0xb71e31b1

Logon Type:         3

Thanks for any insight you can provide

--Mike

Tags (1)
1 Solution

Builder

I believe the third regex from the bottom in your list should work. I got a similar one to match from the search interface with rex (don't have a convenient props.conf to test with at the moment):


rex ".(?<foo>Logon Type:\s+\d)."

That said, the transforms stanzas in props.conf are applied in the order they're listed. In your case, the logon type 3 events match both events-null3 and events-filter. Are you sure directing to nullqueue takes effect and cancels out the events-filter action? Try removing events-filter (or swapping the order, though that may have the same problem) and see if the logon type 3 events are dropped properly.

View solution in original post

Explorer

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out Logon Types after that. But I still had an error with my regex that was causing all Login Types to be filtered out instead of ony those with a value of 3. I finally got that fixed, here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue

Path Finder

In version 6.4 this configuration (props.conf & transforms.conf) needed to be applied on the Universal Forwarder not the indexers. Hopefully that saves some people some time.

SplunkTrust
SplunkTrust

@cam343 - this is only valid for structured data http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Extractfieldsfromfileswithstructureddata#Forw... and not for all kind of data.

Builder

I believe the third regex from the bottom in your list should work. I got a similar one to match from the search interface with rex (don't have a convenient props.conf to test with at the moment):


rex ".(?<foo>Logon Type:\s+\d)."

That said, the transforms stanzas in props.conf are applied in the order they're listed. In your case, the logon type 3 events match both events-null3 and events-filter. Are you sure directing to nullqueue takes effect and cancels out the events-filter action? Try removing events-filter (or swapping the order, though that may have the same problem) and see if the logon type 3 events are dropped properly.

View solution in original post

Explorer

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out event types after that. But I still had an error with my regex which I finally nailed down. Here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!