topic Re: regex working in search but not when used from props/transforms in Splunk Search

Why is regex working in search but not when used from props/transforms?

att35 — Fri, 17 Mar 2023 16:07:10 GMT

We have some logs coming in the following format.

ERROR | 2023-03-16 01:27:14 EDT | field1=field1_value | field2=field2_value | field3=field3_value | field4=field4_value | field5=field5_value | field6=field6_value | field7={} | message=Message String with spaces. java.stacktrace.Exception: Exception Details. at ... at ... at ... at ...

Splunk's default extraction works well in getting all key=value pairs, except for the field "message" where only first word before the space is extracted and drops the rest.

To get around this, I used the following inline regex.

| rex field=_raw "message=(?<message>.+)"

This works well in search and extracts the entire message string right until the newline. But when I used the same regex in the configuration file, it seems to be ignoring the newline and continues to match everything else all the way until end of the event. Have tried using EXTRACT as well as REPORT(using transforms.conf) but same result.

Do props.conf/transforms.conf interpret regex differently?

To summarize,

default Splunk extraction,

message = Message

with inline rex

message = Message String with spaces.

with regex in props/transforms,

message = Message String with spaces. java.stacktrace.Exception: Exception Details. at ... at ... at ... at ...

Any suggestions on how to use this regex from configuration?

Thank you,

Re: regex working in search but not when used from props/transforms

gcusello — Fri, 17 Mar 2023 15:41:47 GMT

Hi @att35,

please try this regex:

(?ms).*message=(?<message>.+)"

Ciao.

Giuseppe

Re: regex working in search but not when used from props/transforms

att35 — Fri, 17 Mar 2023 17:34:12 GMT

Thanks @gcusello

I tried this with EXTRACT. Renaming the extracted field for comparison.

When I used it with quotes, nothing gets extracted. No Msg field

EXTRACT-fullmessage = "(?ms).*message=(?<Msg>.+)"

And when I try without the quotes, extraction works but it does the same thing as before. Entire event from message onwards gets included.

EXTRACT-fullmessage = (?ms).*message=(?<Msg>.+)

Re: regex working in search but not when used from props/transforms

gcusello — Fri, 17 Mar 2023 18:25:39 GMT

Hi @att35,

you don't have to use quotes in props.conf.

It should work as you can see at https://regex101.com/r/EbFNFY/1

Ciao.

Giuseppe

Re: regex working in search but not when used from props/transforms

att35 — Fri, 17 Mar 2023 18:32:40 GMT

What I am trying to extract is the highlighted part.

If I change that regex to

(?m).*message=(?<Msg>.+)

it works, but when used in props behavior is still the same.

So basically, I only want to extract the message string before the Java stack trace starts.

Re: Why is regex working in search but not when used from props/transforms?

somesoni2 — Fri, 17 Mar 2023 18:33:51 GMT

Try this

(?ms).*message=(?<Msg>[^\r\n]+)

Capturing everything till first newline character.

Re: regex working in search but not when used from props/transforms

gcusello — Sat, 18 Mar 2023 07:05:16 GMT

Hi @att35,

what happend if you use (?ms) in the props?

Ciao.

Giuseppe

Re: regex working in search but not when used from props/transforms

PickleRick — Sat, 18 Mar 2023 09:34:37 GMT

All regexes are automatically prepended with (?ms) so you don't have to set it explicitly.

* dotall (?s) and multi-line (?m) modifiers are added in front of the regex.
  So internally, the regex becomes (?ms)<regex>.

Re: regex working in search but not when used from props/transforms

PickleRick — Sat, 18 Mar 2023 09:38:16 GMT

@somesoni2's solution should work. It captures a string _not_ conaining newlines which means that it stops capturing at first encountered newline.

Re: regex working in search but not when used from props/transforms

att35 — Sat, 18 Mar 2023 13:23:33 GMT

It did start the match for message but did not stop at the newline. It continues all the way till end of the event.

Re: regex working in search but not when used from props/transforms

att35 — Sat, 18 Mar 2023 13:24:44 GMT

Yes. That solution worked perfectly. It stopped at the newline and extracted only the characters before that.

Re: Why is regex working in search but not when used from props/transforms?

att35 — Sat, 18 Mar 2023 13:25:33 GMT

Thanks @somesoni2

This did the trick.