topic Re: Can you help me figure out what is the job of the rex field in this line? in Splunk Search

Can you help me figure out what is the job of the rex field in this line?

ramanir — Mon, 31 Dec 2018 07:55:10 GMT

This is the search:

index=vha_pronto sourcetype=pronto_neopil_prd NOT [ search index=vha_pronto sourcetype=pronto_neopil_prd "SAF process started" earliest=-24h |rex field=_raw "(?ms)^[^\\[\\n]*\\[(?P\\w+\\-\\d+)" | return $SAF_pool ]

Re: Can you help me figure out what is the job of the rex field in this line?

ramanir — Tue, 29 Sep 2020 22:33:15 GMT

corrected query

index=vha_pronto sourcetype=pronto_neopil_prd NOT [ search index=vha_pronto sourcetype=pronto_neopil_prd "SAF process started" earliest=-24h |rex field=_raw "(?ms)^[^\[\n]*\[(?P\w+\-\d+)" | return $SAF_pool ]

Re: Can you help me figure out what is the job of the rex field in this line?

richgalloway — Mon, 31 Dec 2018 13:54:28 GMT

The rex command is usually used to extract fields from an event using a regular expression. This rex command is garbled, however, so it's difficult to say precisely what it is doing. Please re-post the query by surrounding it with backticks (`).

Re: Can you help me figure out what is the job of the rex field in this line?

DalJeanis — Mon, 31 Dec 2018 17:33:51 GMT

@ramanir - Be sure to mark your code with the code button (101 010) or by putting at least 4 spaces in front of it. that will stop the interface from stripping out anything that looks like html.

Re: Can you help me figure out what is the job of the rex field in this line?

DalJeanis — Tue, 29 Sep 2020 22:38:01 GMT

1) To answer the exact question you asked: In a rex command, the default field to be analyzed is _raw, so technically, that field=_raw clause simply makes the default explicit, and has no other effect on the function of the rex.

2) The overall search says, "look for events in this index and sourcetype that do not have an SAF process started record in the last 24 hours." The function of the rex is to extract those SAF_pool values from all relevant events in the last 24 hours.

3) Please mark your code when posting. There are three easy methods (A) put grave accents (the one on the ~ key) before and after small snippets of text (b) Put at least four spaces on the line before each line of code, and a blank line before them. (c) highlight the code and press the "code" button (101 010).

My guess is that your rex really reads like this...

 "(?ms)^[^\[\\n]*\[(?P<SAF_pool>\\w+\-\\d+)"

That breaks down as "from the beginning of the line, throw away everything that is not an open brace or carriage return, until you get to an open brace. after that, match one or more word characters, a hyphen, and one or more digits." As such, your SAF_pool numbers are probably in the format ABCD-123 up to and including as weird as AB1_cD-00123. If the prefix is always going to be alpha, then change the \\w+ to [A-Za-z]+.

4) Now here's a breakdown on your entire search code.

First, when there are subsearches, you always read the code from the innermost square braces.

search index=vha_pronto sourcetype=pronto_neopil_prd "SAF process started" 
earliest=-24h 
| rex field=_raw "(?ms)^[^\[\\n]*\[(?P\\w+\-\\d+)" 
| return $SAF_pool

My guess is that the rex really reads like this...

 "(?ms)^[^\[\\n]*\[(?P<SAF_pool>\\w+\-\\d+)"

If I am correct, then what that rex is doing is extracting the SAF_pool information from the events selected by that subsearch. The subsearch brackets will then feed back the answer in a form that looks like this, for all pools started in the last 24 hours...

 ( (  SAF_Pool="ABC-123"  ) OR ( SAF_pool="XYZ-456" ) OR ... )

If you want to know why it turns into that format, look at the documentation for the "format" command.

After returning those values, the rest of the search then looks like this...

index=vha_pronto sourcetype=pronto_neopil_prd  NOT ( (  SAF_Pool="ABC-123"  ) OR ( SAF_pool="XYZ-456" ) OR ... )

Which, as I said before, is basically asking "show me events in an SAF_pool that was started more than 24 hours ago.

Re: Can you help me figure out what is the job of the rex field in this line?

woodcock — Tue, 01 Jan 2019 04:46:22 GMT

Look at the explanation in the upper-right pane here:

https://regex101.com/r/OXBli1/1

Re: Can you help me figure out what is the job of the rex field in this line?

ramanir — Tue, 01 Jan 2019 12:55:00 GMT

thanks a lot for your detailed answer @DalJeanis