topic Re: Subtring from url field and then group using the url in Splunk Search

Subtring from url field and then group using the url

arunprasadlv — Wed, 04 May 2016 20:49:43 GMT

I have a field "BackendURL" which contains different url's.

for eg :

http://abc.com/emp?name=jim&no=101
http://abc.com/emp?name=tim&no=102
http://gef.com/vehicle

I am trying to generate a report by grouping the url's. Now when i group i want to uniquely identify the backend url , but ignore the queury params (anything after ?). I wrote this rex command to create a new field to substring the value before ? , but it does not work when the url does not contain "?" .

index="idx" Consumer|rex field=BackendURL "^(?.+?)\?"|stats count by BackendURL,url_noparams, host

Thanks and regards
Arun

Re: Subtring from url field and then group using the url

sundareshr — Wed, 04 May 2016 21:03:49 GMT

Try this
| eval baseURL=mvindex(split(BackendURL, "?"), o) | stats count by baseURL

Re: Subtring from url field and then group using the url

somesoni2 — Wed, 04 May 2016 21:21:25 GMT

Try like this

index="idx" Consumer|rex field=BackendURL  "(?P<requestedUrl>(?P<path>https*:\/\/((?P<contextRoot>[^\/\s]+)\/)?([^\/\s\?;=]+\/)*)((?P<filename>[^\/\s\?;=]+))?|\-)" |stats count by BackendURL,requestedUrl , host

Re: Subtring from url field and then group using the url

arunprasadlv — Wed, 04 May 2016 23:16:35 GMT

Thanks a Lot. it worked!!

But I am struggling to understand the logic . Would you pls explain

Regards,
Arun

Re: Subtring from url field and then group using the url

somesoni2 — Thu, 05 May 2016 14:09:47 GMT

I'm splitting the url to it's different portions

http://abc.com/emp?name=jim&no=101
|--Path------------------|
          |--contextroot-|
                          |-filename-|

Re: Subtring from url field and then group using the url

arunprasadlv — Thu, 05 May 2016 16:48:10 GMT

Got you. Thanks for the quick reply.

Re: Subtring from url field and then group using the url

donB — Tue, 26 Jan 2021 05:08:40 GMT

@somesoni2 Could you please help if the same url contextPath also has path params?

eg - i can have urls of 2 formats

http://abc.com/emp?name=jim&no=101
http://abc.com/car/ford
http://abc.com/car/tesla

here i need the count of uri paths like below

/car/{id} = 2

/emp = 1 (ignore query params)

Re: Subtring from url field and then group using the url

somesoni2 — Wed, 27 Jan 2021 14:55:50 GMT

Give this a try

| makeresults | eval url="http://abc.com/test/emp?name=jim&no=101 http://abc.com/test/car/ford http://abc.com/test/car/tesla" | table url | makemv url | mvexpand url | eval url=replace(url,"\?","/") | rename "COMMENT" as "Above code generates sample data. Replace it with your query" |rex field=url "(?P<requestedUrl>(?P<path>https*:\/\/((?P<contextRoot>[^\/\s]+)\/)?([^\/\s\?;=]+\/)*)((?P<filename>[^\/\s\?;=]+))?|\-)" | stats count by path