topic Re: rex field not visible and cannot be used in eval in Splunk Search

rex field not visible and cannot be used in eval

maniishpawar — Fri, 17 Nov 2017 14:32:43 GMT

Hi I have this query and trying to do a eval case on the rex field value returned

base
| rex "#TAGRESPONSE.*RESPONSETYPE\:(?.+?)LICENSESTATE" 
| eval code=case('RESPONSETYPE'=="ER51","BATCH", 'RESPONSETYPE'=="ER91","NON-BATCH")
| stats count by code

Its not working, as it cannot see the extracted field RESPONSETYPE.
But when i do stats count by RESPONSETYPE, it works just fine.

Re: rex field not visible and cannot be used in eval

DalJeanis — Fri, 17 Nov 2017 14:50:59 GMT

We've marked your code as code, so that HTML-like attributes will not be deleted by the interface.

Re: rex field not visible and cannot be used in eval

DalJeanis — Fri, 17 Nov 2017 15:03:43 GMT

I'm assuming that the second line looks something like this?

 | rex "#TAGRESPONSE.*RESPONSETYPE\:(?<RESPONSETYPE>.+?)LICENSESTATE"

...and that the underlying _raw data looks something like this...

some stuff #TAGRESPONSE:foobar more stuff  RESPONSETYPE:someresponsetype LICENSESTATE:something else

If so, then what is happening is that your rex is picking up the space after responsetype and before licensestate, and that this code would work (but there's a better way)...

'RESPONSETYPE'=="ER51 "

(notice the space after ER51?)

 | rex "#TAGRESPONSE.*RESPONSETYPE\:(?<RESPONSETYPE>\S+)\s+LICENSESTATE"

This assumes that a responsetype cannot include any spaces. By narrowly typing the responsetype as "things that aren't whitespace (\S) you don't have to make it lazy, it will quit when it gets to the first whitespace, and not include that in the responsecode it is collecting for you.

On the other hand, if the responsecode CAN include spaces, then you do it this way...

 | rex "#TAGRESPONSE.*RESPONSETYPE\:(?<RESPONSETYPE>.+?)\s*LICENSESTATE"

Re: rex field not visible and cannot be used in eval

maniishpawar — Fri, 17 Nov 2017 15:16:15 GMT

here is the actual data.
REQUESTTIME: 2017-11-17 09:28:29 RESPONSETIME: 2017-11-17 09:28:29 RETURNCODE: RATEFOR: CIFG BUSINESSFUNCTION: NBS

REQUESTTIME: 2017-11-17 09:28:29 RESPONSETIME: 2017-11-17 09:28:29 RETURNCODE: 9852 RATEFOR: CIFG BUSINESSFUNCTION: NBS

Re: rex field not visible and cannot be used in eval

maniishpawar — Fri, 17 Nov 2017 15:17:11 GMT

Also I tried to use empty space at the beginning and at trailing of ER51 code, it didnt work in any of those scenarios.

Re: rex field not visible and cannot be used in eval

maniishpawar — Fri, 17 Nov 2017 15:21:15 GMT

oh and you can use RETURNCODE for regex, earlier i have given a dummy field name to post question.

Re: rex field not visible and cannot be used in eval

maniishpawar — Fri, 17 Nov 2017 17:39:01 GMT

its weird, this time I tried space both trailing and leading and it worked. so i used trim and trim worked.
but why rex is adding spaces to the value retrieved ?
what should i change in rex to avoid the space, as if i have 5-10 fields extracted, each will have the trailing and leading space to their values

Re: rex field not visible and cannot be used in eval

wenthold — Fri, 17 Nov 2017 18:28:06 GMT

You have to account for possible trailing spaces in the rex, otherwise if they exist in the source data they will be captured. If you know they the spaces are always in the source data you can write:

| rex "#TAGRESPONSE.*?RESPONSETYPE\:(?<RESPONSETYPE>[^\s]+)\s+?LICENSESTATE"

But if you want account for the spaces possibly being there but account for the possibility that they won't?

| rex "#TAGRESPONSE.*?RESPONSETYPE\:\s*?(?<RESPONSETYPE>.*?)\s*?LICENSESTATE"

The first regex is less computationally expensive but won't capture fields that unless they are formatted properly.