Splunk Search
Highlighted

Duplicate rex field extraction results

Explorer

In my log files there is a field (path = info.message) that has a certain string. I want to extract a part of that string by using "rex". Here is my query:

rex field="info.message" "- (?<pcc>\S{3,4}) "

Rex is doing a great job and extracts the value the way I want, but when I try to view results in a table, I get two values in one row, like that (these results are for three log files):
alt text

Even if I use "dedup pcc" before "table pcc", I still get those duplicate value rows. In logs there is another field called "msg" (path = info.msg) that has the same string as "message", so that can be the cause of this duplication, but my query says that the field should be "info.message", not "info.msg".

Does anyone know what is the issue here and what am I missing out on?
Thanks!!

UPD:
After I removed "field="info.message"" from rex command, the consequent search results came without any duplicates, so this issue has something to do with "field" option in rex.

UPD2:
I have used a workaround with spath and it worked. If you use spath before rex and insert spath's "output" value to rex'es "field", there are no duplicates in results. However, this means that in order to successfully use rex'es "field" option, a user should be aware of spath, which is kind of poor usability, I think.
woodcock, is there a chance a feature request could be added to enable rex'es field value to accept full paths (like in my example - "info.message")? Thank you a lot!

UPD3:
Below is a screenshot of the "info.message" field. I erased confidential info, but that is not important, because I am trying to extract the visible 4 symbol code. As you can see, there are 3 unique records but each of them has a count of 2. Also, there is another field that has a path "info.msg" and contains the exact same information as "info.message".
DalJeanis, do you know maybe whether rex'es field treats its input not as a path but rather a kind of alias or something? If yes, then how can I know that "info.message" is an alias?
alt text

0 Karma
Highlighted

Re: Duplicate rex field extraction results

Esteemed Legend

If your description is accurate then this is definitely a bug so I would certainly open a support case with Splunk.

0 Karma
Highlighted

Re: Duplicate rex field extraction results

Esteemed Legend

I would add a bug tag to this question, too.

0 Karma
Highlighted

Re: Duplicate rex field extraction results

SplunkTrust
SplunkTrust

1) Please post an example of the entire _raw that causes the issue, minus any confidential data, plus any parts of your search that extract the info.message

2) try mvdedup after the field="info.message" option on the rex and see what happens.

0 Karma
Highlighted

Re: Duplicate rex field extraction results

Esteemed Legend

Remember he said then when he operates on _raw, it works fine. If that's true, then it has to be a bug.

0 Karma
Highlighted

Re: Duplicate rex field extraction results

SplunkTrust
SplunkTrust

Check to see if info.message is an alias for another field as well as an extracted field in its own right.

That is, do a search that returns one of the records in question, and look at the record's fields to see if info.message has two values.

0 Karma