rex - multiple matches and using |

[jshaynes]

I'm trying to run several field extractions using the rex command. Here is a sample log format:

ironportmail: Info: MID 42342 ICID 1234 From: xyz@yyx.com
ironportmail: Info: MID 42342 ICID 1234 To: abc@def.com

I'd like to extract the MID, ICID, From and To fields using just one rex command. (This is so I can wrap an eventtype around it.)

Conceptually I'm looking for something like this, but havent been able to get it to work:

rex field=_raw "MID (?[^ ]+) | ICID (?[^ ]+) | From: <(?[^>]+)> | To: <(?[^>]+)"

[Ayn · Accepted solution]

I think you'd be better off putting this in props.conf / transforms.conf instead so you won't have to specify your field extractions explicitly in each search that you want to use these fields.

That said, this will work, though with a caveat:

... | rex field=_raw max_match=10 "(?:MID (?<mid>[^ ]+)|ICID (?<icid>[^ ]+)|From: <(?<mailfrom>[^>]+)>|To: <(?<mailto>[^>]+))"

The caveat being that you will need to provide rex with a max_match (default is 1), otherwise it will stop once it finds its first match.

[jshaynes]

couple of follow up questions:

1. what does the ?: at the beginning do. the query seems to work fine without it
2. any way to prevent all of the empty matches that get returned as a result of changing max_match?

ex, I get a bunch of empty mailfrom, mailto, etc. fields since they don't all occur on every line:

sample field list from search results:
mailfrom="" | mailfrom="" | mailfrom="" | mailfrom=""abc@bvd.com" | mailto="" | mailto="" | mailto="xyz@abc.com" | ... etc

[jshaynes]

awesome, that did it. thanks.

the reason im running these inline is because I am searching off of a summary index. those fields do already get extracted automatically for regular searches, but since the fields get dropped after doing a " | collect index=summary", im having to recreate them.