Multiple Domains

[CyberWolf]

I have an SPLQ that im trying to collect all domains from a raw logs, but my regex is capturing only one domain.
in a single event, some events have one url some of them have 20 and more, how do i capture all domains, please advice?

SPLQ
..............
| rex field=_raw "(?<domain>\w+\.\w+)\/"
| rex field=MessageURLs "\b(?<domain2>(?:http?://|www\.)(?:[0-9a-z-]+\.)+[a-z]{2,63})/?"
| fillnull value=n/a
| stats count by domain domain2 MessageURLs _raw

[CyberWolf]

its a good idea, but in my case i don't know what domains they will be in the _raw, so i cant predict the list. Some events have one domain and it captured but the next event has 5, next will have 12 and each event has different domains on the raw.

[renjith_nair · Accepted solution]

it shouldn't matter. Here is a run anywhere example with multiple events with different number of URL

| makeresults 
| eval _raw="here are some url and http://firsturl.com and some text again url http://www.secondurl.com and again some text URL http://www.third.com/" 
| append 
    [| makeresults 
    | eval _raw="here are some url and http://fourth.com and some text again url"] 
| append 
    [| makeresults 
    | eval _raw="here are some url and http://fifth.com and some text again url and some text again url http://www.sixth.com and again some text http://www.seventh.com and some http://www.moreandmore.com"] 
| rex max_match=0 "\b(?<domain2>(?:http?://|www\.)(?:[0-9a-z-]+\.)+[a-z]{2,63})/?"

[CyberWolf]

Yap it makes since now, it worked thanks!

[renjith_nair]

Try adding the max_match parameter

Here is a run anywhere example.  Please note that the resulted field from  max_match will be a multivalued field in case you want further operation on the field.

| makeresults
| eval _raw="here are some url and http://firsturl.com and some text again url http://www.secondurl.com and again some text URL http://www.third.com/"
| rex max_match=0 "\b(?<domain2>(?:http?://|www\.)(?:[0-9a-z-]+\.)+[a-z]{2,63})/?"