topic rex - multiple matches and using | in Splunk Search

rex - multiple matches and using |

jshaynes — Mon, 12 Dec 2011 23:43:39 GMT

I'm trying to run several field extractions using the rex command. Here is a sample log format:

ironportmail: Info: MID 42342 ICID 1234 From: xyz@yyx.com
ironportmail: Info: MID 42342 ICID 1234 To: abc@def.com

I'd like to extract the MID, ICID, From and To fields using just one rex command. (This is so I can wrap an eventtype around it.)

Conceptually I'm looking for something like this, but havent been able to get it to work:

rex field=_raw "MID (?[^ ]+) | ICID (?[^ ]+) | From: <(?[^>]+)> | To: <(?[^>]+)"

Re: rex - multiple matches and using |

Ayn — Tue, 13 Dec 2011 08:22:25 GMT

I think you'd be better off putting this in props.conf / transforms.conf instead so you won't have to specify your field extractions explicitly in each search that you want to use these fields.

That said, this will work, though with a caveat:

... | rex field=_raw max_match=10 "(?:MID (?<mid>[^ ]+)|ICID (?<icid>[^ ]+)|From: <(?<mailfrom>[^>]+)>|To: <(?<mailto>[^>]+))"

The caveat being that you will need to provide rex with a max_match (default is 1), otherwise it will stop once it finds its first match.

Re: rex - multiple matches and using |

jshaynes — Tue, 13 Dec 2011 18:44:45 GMT

awesome, that did it. thanks.

the reason im running these inline is because I am searching off of a summary index. those fields do already get extracted automatically for regular searches, but since the fields get dropped after doing a " | collect index=summary", im having to recreate them.

Re: rex - multiple matches and using |

jshaynes — Tue, 13 Dec 2011 18:51:27 GMT

couple of follow up questions:

what does the ?: at the beginning do. the query seems to work fine without it
any way to prevent all of the empty matches that get returned as a result of changing max_match?

ex, I get a bunch of empty mailfrom, mailto, etc. fields since they don't all occur on every line: