topic Re: Why does rex/regex return different results than field-extraction? in Splunk Search

Why does rex/regex return different results than field-extraction?

jared_anderson — Tue, 29 Sep 2020 16:53:10 GMT

Data:
Nov 16 12:50:51 172.23.0.29 Nov 16 12:50:51 dc01 Microsoft_Windows_security_auditing.[1688]: Domain\user1: Security Microsoft Windows security auditing.: [Success Audit] A user account was changed.
Subject:
Security ID: domain\value
Account Name: value
Account Domain: Domain
Logon ID: 0xA058EB26
Target Account:
Security ID: domain\user1
Account Name: user1
Account Domain: domain
Changed Attributes:
SAM Account Name: -
Display Name: -
User Principal Name: -
Home Directory: -
Home Drive: -
Script Path: -
Profile Path: -
User Workstations: -
Password Last Set: 11/16/2017 12:50:50 PM
Account Expires: -
Primary Group ID: -
AllowedToDelegateTo: -
Old UAC Value: -
New UAC Value: -
User Account Control: -
User Parameters: -
SID History: -
Logon Hours: -
Additional Information:
Privileges: - (EventID 4738)

Regex Expression: Target Account:\n.+\n Account Name: (?<target_user>.+)

Question:
When I run this regular expression using the rex command it only matches. "user1"

When use this regex in a field extraction it matches everything from user1 to the end of the log. Why does this expression return different results depending on how it is used?

Re: Why does rex/regex return different results than field-extraction?

cpetterborg — Thu, 16 Nov 2017 21:35:46 GMT

It appears that your question needs some tweaking. Can you edit your question, or add a comment, and use the 101010 button on the highlighted "code" to make sure that there are not going to be characters that are removed due to having the code be interpreted before being displayed. Right now the Regex expression doesn't seem to be complete, so it is hard to give you help.

Thanks!

Re: Why does rex/regex return different results than field-extraction?

jared_anderson — Fri, 17 Nov 2017 16:12:35 GMT

Updated, thanks for the help.

Re: Why does rex/regex return different results than field-extraction?

tmarlette — Fri, 17 Nov 2017 16:49:39 GMT

It's likely because there's line ending characters that you can't see (printed by MS) on some events, and then sometimes not on others. regex is VERY specific about things, even the ones you can't see.

If you use a '.+' it's a lazy match, so just try to be more specific with your anchor characters. MS is not known for it's ease of matching in it's events. They are quite verbose.

Re: Why does rex/regex return different results than field-extraction?

jared_anderson — Fri, 17 Nov 2017 16:54:22 GMT

I don't understand why it would act differently using the rex command vs an extracted field in the same log.

Re: Why does rex/regex return different results than field-extraction?

cpetterborg — Fri, 17 Nov 2017 16:58:51 GMT

I don't know if it is a typo, but your REGEX has an extra space before Account. It should be:

Target Account:\n.+\nAccount Name: (?<target_user>.+)

Like I say, it may be just a typo, but it would not match as you have supplied it.

Re: Why does rex/regex return different results than field-extraction?

jared_anderson — Fri, 17 Nov 2017 17:16:01 GMT

Formatting the submission removed the leading spaces. There is actually a space before Account Name in the real logs.

I just checked again. With a field extraction, I am getting a match, but the match includes from user1 to the end of the log. If I take the same expression and use the rex command in search, I get exactly what I want.

Re: Why does rex/regex return different results than field-extraction?

wenthold — Fri, 17 Nov 2017 19:47:33 GMT

Are you using the EXTRACT- in props.conf to do the extraction? By default the multiline regex modifier is enabled when doing field extraction that way.

props.conf - field extraction

Re: Why does rex/regex return different results than field-extraction?

mtulett_splunk — Tue, 21 Nov 2017 05:55:09 GMT

@wenthold is correct - the field extraction engine is adding the ?ms flags to the start of the regex in your props.conf file. This means that your regex is matching on the whole event as though it's a single line. This can be seen in the documentation too:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf?utm_source=answers&utm_medium=in-answer&utm_term=props.conf&utm_campaign=refdoc#Field_extraction_configuration

If your aim is to only extract the target account, I'd recommend using something like the following regex, which will work in field extractions or in the rex command:

(?m)Target\sAccount:\n?(?:(?!Changed).+?)\n?Account\sName:\s(?<target_user>.+?)\n?Account

This will also block capture of the rest of the event if the field is empty or all on one line, which is what's happening for you at the moment.