Regex transforms not applied on Windows event logs

[jkalbert]

I wish to remove unneeded text from Windows event logs before they are indexed. Specifically, Windows event 4624 contains a dozen or so lines of text at the end that I don't want. These are sent to my clustered indexers directly from universal forwarders.

I have a transform set up on the indexers to extract the fields I'm interested in:

[events-win-security-4624]
REGEX = EventCode=(4624)\n.*\nComputerName=(.*)?\n.*\n.*\n.*\n.*\n.*?\n.*\n.*\n\nSubject:\n.*\n*.*\n*.*\n*.*\n\nLogon Information:\n*.*\n*.*\n*.*\n*.*\n\n\n*.*\n\nNew Logon:\n\s*Security ID:\s*(.*)?\n\s*Account Name:\s*(.*)?\n\s*Account Domain:\s*(.*)?\n.*\n.*\n.*\n.*\n.*\n\n.*\n.*\n.*\n\nNetwork Information:\n\s*Workstation Name:\s*(.*)?\n\s*Source Network Address:\s*(.*)?\n\s*Source Port:\s*(.*)?\n\nDetailed Authentication Information:\n.*Logon Process:\s*(.*)?
DEST_KEY = _raw
FORMAT = EventCode=$1 ComputerName=$2 SecurityID=$3 AccountName=$4 AccountDomain=$5 WorkstationName=$6 SourceNetworkAddress=$7 SourcePort=$8 LogonProcess=$9

(Please pardon the lengthy regex; all it's doing is capturing the relevant fields.)

And the corresponding bit from props.conf, also on the indexers:

[WinEventLog:Security]
TRANSFORMS-winsec_event4624 = events-win-security-4624

Based on my understanding, the transform should extract the specified fields from the raw event, put these into the format specified by the FORMAT line, send it on to be indexed, and ignore the rest.

However, this is not happening. When I search my index for "EventCode=4624" I see the full event text, along with the extraneous text. It doesn't appear that the transform is doing anything.

I've tested my regex against multiple events using regex101.com and everything looks correct.

I've since set up a few other regex transforms to drop specific events from a different source (by sending them to nullQueue) and these are all working as expected, so I know that Splunk is able to see the conf files.

I've reviewed the specifications for transforms.conf and props.conf but wasn't able to find what I'm doing wrong.

Am I going about this the correct way? I'm relatively new to Splunk so I'm sure I'm overlooking something simple.

Thank you for your help!

[richgalloway · Accepted solution]

This is how the Windows TA (https://splunkbase.splunk.com/app/742) does that.  If you have the TA then just uncomment this line and restart Splunk.

SEDCMD-clean_info_text_from_winsystem_events_this_event = s/This [Ee]vent is generated[\S\s\r\n]+$//g

If you want to keep what you have then the sequence [\s\S]+ works best when trying to match any text over several lines.

[jkalbert]

Thank you, the sed solution worked! I added it to my props.conf under the [WinEventLog:Security] stanza and now that extraneous text is being removed from events.