topic Re: Regex in Splunk Enterprise

Regex

praddasg — Wed, 03 Mar 2021 00:51:24 GMT

Hello All,

I am not so familiar with regex, but looking at some old query have been able to build one for my need. I am looking for help to understand how this is working in terms of regular expression and Splunk rex syntax

So the regex I am using is

| rex field=_raw message="(?<message>.*).request"

for the

message=abc ff request-id

where I am trying to extract anything after "=" until "request-id". There could be spaces as well

I think "<message>" here is the field name I want to denote
The wild card character "*" within the braces indicate everything after "message="
But I don't understand
1. The use of "?". Is this part of the syntax of splunk regex or signifying anything and everything after "message=" i.e. working along with "*"
2. What is the use of braces here? is this indicating the section I am trying to parse?
3. The dot "." after "<message>". Is this splunk syntax?
4. The dot "." after braces. Is this denoting/delimiting/indicating the string which is present after the parsing section
5. The most confusing part is the use of quotes.
6. What would be regex if it is like "message abc ff request-id" and I want to parse anything between message and request

Re: Regex

richgalloway — Wed, 03 Mar 2021 01:07:01 GMT

The example rex command is invalid. The regular expression must be enclosed in quotation marks, like this

| rex field=_raw "message="(?<message>.*).request""

then the embedded quotation marks must be escaped, like this

| rex field=_raw "message=\\\"(?<message>.*).request\\\""

1. <message> denotes the name of the capture group and is the name of the field the matching text will fill.

2. The regex wildcard character is . (full stop). The asterisk (*) is a quantifier that means "any number of these". The sequence .* ("dot-star") means "everything from here on".

3a. The "?" means nothing by itself in this context. The "(?" sequence starts a capture group.

b. There are no braces in this regex so do you mean the parentheses or the angle brackets? In this context, the parentheses denote a capture group and the angle brackets denote the name of the current capture group. In Splunk, this becomes a field name.

c. Like mentioned in 2 above, the dot is the wildcard character. Is it standard regex, not specific to Splunk.

d. See 2 and c.

e. Quotation marks are not special characters in regex. They're just another character to match. On the other hand, embedded quotation marks in the rex command ARE confusing. They require 3 escape characters to get through the various parsers to the regex engine.

f. Something like "message (.*) request".

If you pass a regex string into https://regex101.com the site will explain what each character means.

Re: Regex

praddasg — Wed, 03 Mar 2021 14:47:26 GMT

Hello @richgalloway

Thank you for taking the time and explaining. I really appreciate the time you vested in explaining this.

Interestingly this one works

| rex field=_raw message="(?<message>.*).request"

So does the

| rex field=_raw "message="(?<message>.*).request""

but not the

"message=\\\"(?<message>.*).request\\\""

when I say work, I mean it is giving the desired result and by not working I mean not giving the desired result. Although in none of the cases there wasn't any syntax error.

The one with the escaped quotation mark only gives the result until before the spaces i.e. if it is "message=abc efg request-id", it only prints "abc". Does this have anything to do with the Splunk version?

2. Regarding The sequence .* ("dot-star") means "everything from here on" - I am assuming this regex and nothing to do with Splunk itself. So I tried to use this concept in a sublime text editor to see what happens. I used

message=Error translating Grubhub webhook order: The location for this order cannot be found request-id

and tried to replace message=.* with let's say new. I found the entire thing got wiped out and replaced with new. I was expecting something like message=new. I even tried message="(?.*).request", "message="(?.*).request"", but no changes happened. Is it because Splunk uses some different regex logic than sublime text editor?

3. I am still confused about the use of quotation mark, I tried using the website which you mentioned, but it confused me more lol.

Re: Regex

richgalloway — Wed, 03 Mar 2021 15:50:26 GMT

To know whether a regex works or not requires knowing the text it is trying to match. Please share.

I don't know which regex version Sublime uses, but Splunk uses PCRE. Also, Splunk is not a text editor so it may behave differently from an editor.

You may find this helpful: https://conf.splunk.com/files/2017/slides/regex-in-your-spl.pdf

Re: Regex

praddasg — Wed, 03 Mar 2021 16:06:30 GMT

@richgalloway the text trying to match here is anything after "=", until "request" so the complete text here is

message=abc ef x request-id

Re: Regex

praddasg — Wed, 03 Mar 2021 16:11:21 GMT

oh one more thing, the content between "=" and "request" could be any number of character or number and can have multiple spaces as well

Re: Regex

richgalloway — Wed, 03 Mar 2021 17:52:42 GMT

"=.*?request"

The question mark limits the scope of the asterisk to the fewest number of characters needed to match the regex.