Splunk Search

Why is regex working in search but not when used from props/transforms?

att35
Builder

We have some logs coming in the following format. 

 

ERROR | 2023-03-16 01:27:14 EDT | field1=field1_value | field2=field2_value | field3=field3_value | field4=field4_value | field5=field5_value | field6=field6_value | field7={} | message=Message String with spaces.
java.stacktrace.Exception: Exception Details.
	at ...
	at ...
	at ...
	at ...

 

 

Splunk's default extraction works well in getting all key=value pairs, except for the field "message" where only first word before the space is extracted  and drops the rest.

To get around this, I used the following inline regex.

 

|  rex field=_raw "message=(?<message>.+)"

 

This works well in search and extracts the entire message string right until the newline. But when I used the same regex in the configuration file, it seems to be ignoring the newline and continues to match everything else all the way until end of the event. Have tried using EXTRACT as well as REPORT(using transforms.conf) but same result.

Do props.conf/transforms.conf interpret regex differently?

To summarize, 

default Splunk extraction,

 

 message = Message

 

with inline rex

 

message = Message String with spaces.

 

with regex in props/transforms, 

 

message = Message String with spaces.
java.stacktrace.Exception: Exception Details.
at ...
at ...
at ...
at ...

 

 

Any suggestions on how to use this regex from configuration? 

Thank you,

Labels (3)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try this

 

(?ms).*message=(?<Msg>[^\r\n]+)

 Capturing everything till first newline character. 

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Try this

 

(?ms).*message=(?<Msg>[^\r\n]+)

 Capturing everything till first newline character. 

att35
Builder

Thanks @somesoni2 

This did the trick.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @att35,

please try this regex:

(?ms).*message=(?<message>.+)"

Ciao.

Giuseppe

0 Karma

att35
Builder

Thanks @gcusello 

I tried this with EXTRACT.  Renaming the extracted field for comparison.

When I used it with quotes, nothing gets extracted.  No Msg field

EXTRACT-fullmessage = "(?ms).*message=(?<Msg>.+)"

 And when I try without the quotes, extraction works but it does the same thing as before. Entire event from message onwards gets included. 

EXTRACT-fullmessage = (?ms).*message=(?<Msg>.+)

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @att35,

you don't have to use quotes in props.conf.

It should work as you can see at https://regex101.com/r/EbFNFY/1

Ciao.

Giuseppe

0 Karma

att35
Builder

What I am trying to extract is the highlighted part.  

 

Splunk_regex_1.png

If I change that regex to 

(?m).*message=(?<Msg>.+)

it works, but when used in props behavior is still the same.  

 

Splunk_regex_2.png

So basically, I only want to extract the message string before the Java stack trace starts.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

@somesoni2's solution should work. It captures a string _not_ conaining newlines which means that it stops capturing at first encountered newline.

0 Karma

att35
Builder

Yes.  That solution worked perfectly. It stopped at the newline and extracted only the characters before that.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @att35,

what happend if you use (?ms) in the props?

Ciao.

Giuseppe

0 Karma

att35
Builder

It did start the match for message but did not stop at the newline. It continues all the way till end of the event. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

All regexes are automatically prepended with (?ms) so you don't have to set it explicitly.

* dotall (?s) and multi-line (?m) modifiers are added in front of the regex.
  So internally, the regex becomes (?ms)<regex>.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...