Multiple rex commands no longer works in Fast/Smart mode?

[mattiaslindblom]

In Splunk 6.6.1, it seems like multiple rex commands with the same field name does no longer work in Fast or Smart mode, if it's followed by for example stats or table.

I want to rex the words "please", "extract" and "me":

index=test please_extract_me
| rex "(?<my_field>.*)_extract_me"
| rex "please_(?<my_field>.*)_me"
| rex "please_extract_(?<my_field>.*)"
| stats count by my_field

Usually, regardless of what search mode was used, I would/should get three lines as result. Now I have to select Verbose mode to get it to work; in Fast or Smart mode, only the last rex works and will show up in the count.

One way around this is apparently by doing this, as if the different rex's would create multiple fields with the same name instead of one field with multiple values:

index=test please_extract_me
| rex "(?<my_field>.*)_extract_me"
| rex "please_(?<my_field>.*)_me"
| rex "please_extract_(?<my_field>.*)"
| fields *
| stats count by my_field

Anyone else experiencing this?

[drodman29]

Support is stating that there is a fix in the works for this with a release soon.
Until then, there are two possible options
1) Do not have more than 1 REX for the same field in your search, extract unique field names and then eval a new name at the end along this design pattern: | eval My_Field=if(isnull(my_field1),my_field2, myfield1)

2) The second is a work around to turn off optimizations by placing this at the end of your SPL
| noop search_optimization=f

Both of these options are working in my 7.0 install.

[mschaaf]

Do you happen to know the Issue Number? I didn't see anything relevant under KnownIssues for the versions where I'm experiencing this.

[drodman29]

My case # was 577972 - At the time I was running Splunk Build c8a78efdd40f for 7.0.0

[RMartinezDTV]

Just so the OP doesn't think he's going crazy, I hit this same bug as well after today's upgrade (v6.6.1). Everything works OK when running in Verbose mode which is (to my knowledge) not an option for dashboard panels.

EDIT: I see the workaround I found has already been posted in the comment thread, but I'm repeating it here as an "Answer" for the public's sake.

To workaround this issue, add

| fields *

at the end of the search. I made it the final command before the table and everything works OK now.

[drodman29]

Hit this same issue in 7.0, all search modes, workaround did not work either.

[Shephali]

Has an issue/case already raised for this bug in v6.6.1?
The issue is prominent in dashboards, as Verbose mode is not an option for dashboard panels.

[mattiaslindblom]

Thanks for at least validating my findings.

[MuS]

I agree with all the previous comments, this should not have worked this way.
The only thing I can think of is, that since you're using a string in your base search there were other strings in the events which happen to match the regex as well.

This run everywhere (smart mode) search returns three times the same value my_field=me:

 | gentimes start=-1 
 | eval foo="please_extract_me please_extract_me please_extract_me" 
 | makemv foo 
 | mvexpand foo 
 | rex field=foo "(?<my_field>.*)_extract_me" 
 | rex field=foo "please_(?<my_field>.*)_me" 
 | rex field=foo "please_extract_(?<my_field>.*)"

where as this search returns four different values for my_field and would match your base search because it has please_extract_me:

| gentimes start=-1 
| eval foo="please_extract_me one_extract_me please_two_me please_extract_three" 
| makemv foo 
| mvexpand foo 
| rex field=foo "(?<my_field>.*)_extract_me" 
| rex field=foo "please_(?<my_field>.*)_me" 
| rex field=foo "please_extract_(?<my_field>.*)"

This is just to explain what I meant. I would suggest to add some raw sample events and your real world dashboard search, this way we all can be of more help. Also, are there any calculated fields or lookups happening for the events you get back from the base search?

cheers, MuS

[RMartinezDTV]

Some good discussion here. I hit the exact same issue, however, using the max_match= function of the rex command as opposed to having separate rex commands.

I can understand and agree with the points above when using multiple rex (the field gets overwritten), but I am seeing with same behavior using only a single rex command and the provided "max_match" parameter. That's making me lean closer towards a regression bug as opposed to "previously broken" behavior.

[mattiaslindblom]

The max_match option creates multivalue fields, though, which it should according to the manual. So the behavior you are describing is expected. But this behavior now also seems to apply to multi-line rex. Actually, I just found that adding a | mvexpand field_name after the rex commands solves my problem.

Maybe this was always the intended behavior (even though it obviously hasn't been), but I still find it odd that there's a difference between the search modes.

[renatobamorim]

Did you try use one ony rex command?

...
| rex field=_raw "(?P<field1>[^_])_(?P<field2>[^_])_(?P<field3>.*)"

[mattiaslindblom]

That would create three different fields, though, and I need one field with different values. And for searches where there are different events that you want to rex parts from to the same fields, a one-liner just wouldn't work.

[DalJeanis]

That's not my understanding of how it ever worked... so if it DOES work in verbose mode, that is probably a bug in verbose mode. Each rex should overwrite the field... I have rexes in production that assume this, and that are working as designed.

[woodcock]

A later failed-to-match rex has never overwritten an earlier succeeded-to-mach rex, in my experience. Perhaps this is what he means (that a later null-match is clearing an earlier match).

[mattiaslindblom]

But I know for a fact that it has actually worked before, since I've built functioning dashboards around this behavior, that now don't work any longer.

And again, adding fields * after the multiple rex commands makes it work, which to me doesn't make sense.

[somesoni2]

You're using the same field name in all your rex commands, so every rex command is overwriting the value from previous rex command hence you would see value me that was extracted from last regex. That's how rex behaves regardless of Search mode selected (as seen in Splunk 6.2, 6.3). Do you really get multiple field values from your query when you run the search in Verbose mode?

[mattiaslindblom]

Yes, Verbose mode works fine, and I've been using this in a dashboard that now suddenly has stopped working.

[somesoni2]

That is really strange. Would you be able to share a screenshot with results of query in verbose mode something like this?

index=test please_extract_me| head 2
 | rex "(?<my_field>.*)_extract_me"
 | rex "please_(?<my_field>.*)_me"
 | rex "please_extract_(?<my_field>.*)"
 | table my_field

AFAIK, The dashboards queries are run in fast mode, so wonder it ever worked.

[woodcock]

Add the bug tag and open a support case. Also go to the documentation page for the release notes and leave a comment there that a note should be added.

[woodcock]

My comment is only valid assuming that you are entirely correct about the change in behavior, which may not be the case.