topic Re: Duplicate rex field extraction results in Splunk Search

Duplicate rex field extraction results

funghorn — Fri, 26 May 2017 12:00:52 GMT

In my log files there is a field (path = info.message) that has a certain string. I want to extract a part of that string by using "rex". Here is my query:

rex field="info.message" "- (?<pcc>\S{3,4}) "

Rex is doing a great job and extracts the value the way I want, but when I try to view results in a table, I get two values in one row, like that (these results are for three log files):

Even if I use "dedup pcc" before "table pcc", I still get those duplicate value rows. In logs there is another field called "msg" (path = info.msg) that has the same string as "message", so that can be the cause of this duplication, but my query says that the field should be "info.message", not "info.msg".

Does anyone know what is the issue here and what am I missing out on?
Thanks!!

UPD:
After I removed "field="info.message"" from rex command, the consequent search results came without any duplicates, so this issue has something to do with "field" option in rex.

UPD2:
I have used a workaround with spath and it worked. If you use spath before rex and insert spath's "output" value to rex'es "field", there are no duplicates in results. However, this means that in order to successfully use rex'es "field" option, a user should be aware of spath, which is kind of poor usability, I think.
woodcock, is there a chance a feature request could be added to enable rex'es field value to accept full paths (like in my example - "info.message")? Thank you a lot!

UPD3:
Below is a screenshot of the "info.message" field. I erased confidential info, but that is not important, because I am trying to extract the visible 4 symbol code. As you can see, there are 3 unique records but each of them has a count of 2. Also, there is another field that has a path "info.msg" and contains the exact same information as "info.message".
DalJeanis, do you know maybe whether rex'es field treats its input not as a path but rather a kind of alias or something? If yes, then how can I know that "info.message" is an alias?

Re: Duplicate rex field extraction results

woodcock — Fri, 26 May 2017 15:28:41 GMT

If your description is accurate then this is definitely a bug so I would certainly open a support case with Splunk.

Re: Duplicate rex field extraction results

DalJeanis — Fri, 26 May 2017 16:03:38 GMT

1) Please post an example of the entire _raw that causes the issue, minus any confidential data, plus any parts of your search that extract the info.message

2) try mvdedup after the field="info.message" option on the rex and see what happens.

Re: Duplicate rex field extraction results

woodcock — Fri, 26 May 2017 16:12:00 GMT

Remember he said then when he operates on _raw, it works fine. If that's true, then it has to be a bug.

Re: Duplicate rex field extraction results

woodcock — Mon, 29 May 2017 14:36:00 GMT

I would add a bug tag to this question, too.

Re: Duplicate rex field extraction results

DalJeanis — Tue, 30 May 2017 16:02:12 GMT

Check to see if info.message is an alias for another field as well as an extracted field in its own right.

That is, do a search that returns one of the records in question, and look at the record's fields to see if info.message has two values.