Why is regex working in search but not when used from props/transforms?

[att35]

We have some logs coming in the following format. 

ERROR | 2023-03-16 01:27:14 EDT | field1=field1_value | field2=field2_value | field3=field3_value | field4=field4_value | field5=field5_value | field6=field6_value | field7={} | message=Message String with spaces.
java.stacktrace.Exception: Exception Details.
	at ...
	at ...
	at ...
	at ...

Splunk's default extraction works well in getting all key=value pairs, except for the field "message" where only first word before the space is extracted  and drops the rest.

To get around this, I used the following inline regex.

|  rex field=_raw "message=(?<message>.+)"

This works well in search and extracts the entire message string right until the newline. But when I used the same regex in the configuration file, it seems to be ignoring the newline and continues to match everything else all the way until end of the event. Have tried using EXTRACT as well as REPORT(using transforms.conf) but same result.

Do props.conf/transforms.conf interpret regex differently?

To summarize, 

default Splunk extraction,

 message = Message

with inline rex

message = Message String with spaces.

with regex in props/transforms, 

message = Message String with spaces.
java.stacktrace.Exception: Exception Details.
at ...
at ...
at ...
at ...

Any suggestions on how to use this regex from configuration? 

Thank you,

[somesoni2 · Accepted solution]

Try this

(?ms).*message=(?<Msg>[^\r\n]+)

 Capturing everything till first newline character. 

[att35]

Thanks @somesoni2 

This did the trick.

[gcusello]

Hi @att35,

please try this regex:

(?ms).*message=(?<message>.+)"

Ciao.

Giuseppe

[att35]

Thanks @gcusello 

I tried this with EXTRACT.  Renaming the extracted field for comparison.

When I used it with quotes, nothing gets extracted.  No Msg field

EXTRACT-fullmessage = "(?ms).*message=(?<Msg>.+)"

 And when I try without the quotes, extraction works but it does the same thing as before. Entire event from message onwards gets included. 

EXTRACT-fullmessage = (?ms).*message=(?<Msg>.+)

[gcusello]

Hi @att35,

you don't have to use quotes in props.conf.

It should work as you can see at https://regex101.com/r/EbFNFY/1

Ciao.

Giuseppe

[att35]

What I am trying to extract is the highlighted part.  

If I change that regex to 

(?m).*message=(?<Msg>.+)

it works, but when used in props behavior is still the same.  

So basically, I only want to extract the message string before the Java stack trace starts.

[PickleRick]

@somesoni2's solution should work. It captures a string _not_ conaining newlines which means that it stops capturing at first encountered newline.

[att35]

Yes.  That solution worked perfectly. It stopped at the newline and extracted only the characters before that.

[gcusello]

Hi @att35,

what happend if you use (?ms) in the props?

Ciao.

Giuseppe

[att35]

It did start the match for message but did not stop at the newline. It continues all the way till end of the event. 

[PickleRick]

All regexes are automatically prepended with (?ms) so you don't have to set it explicitly.

* dotall (?s) and multi-line (?m) modifiers are added in front of the regex.
  So internally, the regex becomes (?ms)<regex>.