topic Re: Reqex filter in transforms.conf not working in Getting Data In

Reqex filter in transforms.conf not working

working_dog — Mon, 02 Dec 2013 14:35:29 GMT

At the indexer, I am trying to exclude event records from incoming windows logs that have Logon Type=3. Below is the configuration that I have, but doesn't seem to work.

$SPLUNK_HOME/etc/system/local/props.conf

[WinEventLog:Security]    
TRANSFORMS-security= events-null, events-null3, events-filter

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-null3]
REGEX=Logon Type=\s*(3)\D
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue

The [events-filter] stanza works. I only see the events listed. The [events-null3] doesn't work. I've tried a variety of regex variations. Many came from different questions posted on this forum. Here are the ones I've tried:

REGEX=Logon_Type=\s*(3)\D
REGEX=Logon Type=\s*(3)\D
REGEX=Logon_Type=\s*(3)
REGEX=(?msi).Logon\sType:\s3D
REGEX=(?msi).Logon\sType:\s3
REGEX=(?m).*Logon\sType:\s+3.*
REGEX=(?m).Logon\sType:\s+3
REGEX = (?msi).*Logon Type:\s*(2|10)

Below is a sample of an event I am trying to filter out.

12/02/2013 08:45:43 AM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4634
EventType=0
Type=Information
ComputerName=P-UMS03.p-umhs.med.umich.edu
TaskCategory=Logoff
OpCode=Info
RecordNumber=5744631540
Keywords=Audit Success
Message=An account was logged off.

Subject:
Security ID:        P-UMS\USPCNTBSA1$
Account Name:       USPCNTBSA1$
Account Domain:     P-UMS
Logon ID:       0xb71e31b1

Logon Type:         3

Thanks for any insight you can provide

--Mike

Re: Reqex filter in transforms.conf not working

cphair — Mon, 02 Dec 2013 15:44:21 GMT

I believe the third regex from the bottom in your list should work. I got a similar one to match from the search interface with rex (don't have a convenient props.conf to test with at the moment):



rex ".(?<foo>Logon Type:\s+\d)."

That said, the transforms stanzas in props.conf are applied in the order they're listed. In your case, the logon type 3 events match both events-null3 and events-filter. Are you sure directing to nullqueue takes effect and cancels out the events-filter action? Try removing events-filter (or swapping the order, though that may have the same problem) and see if the logon type 3 events are dropped properly.

Re: Reqex filter in transforms.conf not working

working_dog — Mon, 02 Dec 2013 20:14:05 GMT

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out event types after that. But I still had an error with my regex which I finally nailed down. Here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue

Re: Reqex filter in transforms.conf not working

working_dog — Mon, 02 Dec 2013 20:39:36 GMT

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out Logon Types after that. But I still had an error with my regex that was causing all Login Types to be filtered out instead of ony those with a value of 3. I finally got that fixed, here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue

Re: Reqex filter in transforms.conf not working

cam343 — Mon, 16 May 2016 03:04:50 GMT

In version 6.4 this configuration (props.conf & transforms.conf) needed to be applied on the Universal Forwarder not the indexers. Hopefully that saves some people some time.

Re: Reqex filter in transforms.conf not working

MuS — Mon, 16 May 2016 03:25:20 GMT

@cam343 - this is only valid for structured data http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Extractfieldsfromfileswithstructureddata#Forward_data_extracted_from_structured_data_files and not for all kind of data.