topic Re: How to group URLS based patterns? in Splunk Search

How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 00:58:40 GMT

I have n of log files and i'm getting the proper result for each URL as of now, but im facing issue since the same url can be access by the same employee not sure what is the best way i can do it.
Log files :

index 1
baseURL/employees
baseURL/employees/{id}

index 2
baseURL/employees/{id}/comments
baseURL/employees/{id}/comments/{id}

index 2
baseURL/employees/{id}/messages
baseURL/employees/{id}/message/{id}

Re: How to group URLS based patterns?

niketn — Wed, 12 Apr 2017 02:42:28 GMT

@jw44250... You would need to add some sample mock data and your existing metadata(field names)/existing query for us to come up with exact answer.. However, you can use the following in your case, assuming url is the field containing URLs in your log (if not you would need to perform Field Extraction using Splunk's Interactive Field Extraction or rex command):

<YourBaseSearchWithIndexSourceType>
| stats count(eval(match(url,"^baseURL/employees/\d+$"))) as EmployeeAccessCount count(eval(match(url,"^baseURL/employees/\d+/comments/\d+$"))) as CommentsAccessCount count(eval(match(url,"^baseURL/employees/\d+/messages/\d+$"))) as MessagesAccessCount

I have tested with following run anywhere search (you can change url and match condition as per your exact log data/field):

| makeresults
| eval url="baseURL/employees/12345/message/12345"
| table url
| stats count(eval(match(url,"^baseURL/employees/\d+/message/\d+$"))) as MessageAccessCount

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 04:46:56 GMT

Since i have httpRequestURL as key in log files i am getting result i am looking for but i want group them in such away after main urls:

below example : matching employee with 100 and 800 are accessing comments url
localhost/employees/100/comments
localhost/employees/800/comments

matching 600 and 900 id having 3 messages
localhost/employees/600/messages/3
localhost/employees/900/messages/3

httpRequest
localhost/employees/100/comments
localhost/employees/200/comments/10
localhost/employees/300/messages
localhost/employees/400/message/3
localhost/employees/800/comments
localhost/employees/700/comments/10
localhost/employees/600/messages/3
baseURL/employees/400/message/3

Re: How to group URLS based patterns?

martin_mueller — Wed, 12 Apr 2017 07:46:24 GMT

Are you trying to group URLs while ignoring /{id} segments? If so, try this:

... | eval grouped_url = replace(httpRequestURL, "/\d+", "") | stats count by grouped_url

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 15:16:27 GMT

i want to group by comments, employees,

Like Level 1, Level 2, Level 3

More Example

URI HTTP METHOD DESCRIPTION
/employee/add POST Add an employee
/employee/getDummy GET returns a dummy employee object
/employee/{id}/get GET Get the employee with ‘id’ in the URI
/employee/getAll GET Get all employees
/employee/{id}/delete DELETE Delete employee with ‘id’ in the URI

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 15:17:38 GMT

Yes i want to ignore /{id} yes..

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 15:33:17 GMT

i have 1000 of them with different paths

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 15:41:12 GMT

grouped_url = /api/v1/appName/100/message/groups343434

How exclude things after id : grouped_url= /api/v1/appName/100

Re: How to group URLS based patterns?

jw44250 — Wed, 12 Apr 2017 21:36:32 GMT

it does not working

Re: How to group URLS based patterns?

niketn — Thu, 13 Apr 2017 04:52:35 GMT

@jw44250, your questions/requirements seems to be changing.
Since you have different types of URIs, I still expect that you should perform a match on URI with values like messages, comments, employees for you to come up with count etc. (you need to come up with cases based on your data):

eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
| table URI URIType

If you want to just remove digits and use the remaining URI, I would use martin_mueller's answer.

If you want to exclude things after ID like you have asked last, you can try: | eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")

Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")
| table URI requiredURI

On a different approach you can split URIs by forward slash character /. Then use mvindex to read what you are interested in. Like comments in the example below:

| eval arrURI=split(URI,"/")
| table URI arrURI
| search arrURI="comments"
| eval baseURI=mvindex(arrURI,1)
| eval commentNumURI=mvindex(arrURI,4)
| table URI arrURI baseURI commentNumURI

Re: How to group URLS based patterns?

jw44250 — Thu, 13 Apr 2017 05:49:54 GMT

thanks since im analyzing lots of logs file i mean 100s of them.
so getting urls like

URLS : - /1/ or /3/hello .

/20012/users

Re: How to group URLS based patterns?

niketn — Thu, 13 Apr 2017 05:58:55 GMT

@jw44250... You need to post URLs with code button ie. 10101010 so that it does not get omitted while posting. Please re-post URLs with the same.

PS: If my answers above do not cater to your needs, I can convert my answer to comment. However, the same will still not flag this question as unanswered question for all Splunker's. Splunk can definitely handle what you are trying to achieve also if @martin_mueller is answering a question you can be assured that it is the best solution 🙂

So may be simplify your question and ask what you really need.

Re: How to group URLS based patterns?

jw44250 — Tue, 29 Sep 2020 13:40:00 GMT

Excluding following urls using searchmatch

/xampp/zenario/
/xmb/
/xmldata
/xmlrpc.php
/xmlrpc/
/xmlrpcs.inc
/xoops/
/xoops/userinfo.php
/xoops_v2_rc3/html/
/xyzabc.jsp
/y4icpgtwvy0/
/yabb/
/yabbse-1.4.1/
/yabbse/
/yapig-0.92b/
/yapig-0.95b/
/yapig/
/yfPxl446.php4
/yfo267oib0m/
/ymGVHWTx.php
/z24d7fkor/
/zabbix/
/zapbook.cgi
/zapbook/
/zen-cart/
/zenario/
/zenariocms/
/zencart/
/200/
/1/
/2/
/600/anc
/zorum/
/zpanel/
/zsh

Re: How to group URLS based patterns?

jw44250 — Thu, 13 Apr 2017 07:14:46 GMT

but still splunk returns of URLS even i didnt ask for it...using case and searchmatch
it should return only that matching URL below why it returns other URLS which i did not ask for it
eval grouped_url=case(searchmatch("/api/v7/service/*"), "cloudsite.com",1=1,requestURI)

Re: How to group URLS based patterns?

jw44250 — Thu, 13 Apr 2017 16:24:44 GMT

How to replace localhost/employees/100/comments with digit using eval with searchMax

Re: How to group URLS based patterns?

niketn — Thu, 13 Apr 2017 16:38:28 GMT

For your previous request for excluding list of urls using searchmatch, can you provide one full url for each as an example?

/xampp/zenario/
/xmb/
/xmldata
....
....

Re: How to group URLS based patterns?

niketn — Thu, 13 Apr 2017 16:39:06 GMT

For your last question can you let me know what is the output you want when digits are replaced? The reason why I am asking is because martin_mueller has already given you a query to remove digits from URL. So please provide the expected output for the following:
input= localhost/employees/100/comments
output=?