How to group URLS based patterns?

[jw44250]

I have n of log files and i'm getting the proper result for each URL as of now, but im facing issue since the same url can be access by the same employee not sure what is the best way i can do it.
Log files :

index 1
baseURL/employees
baseURL/employees/{id}

index 2
baseURL/employees/{id}/comments
baseURL/employees/{id}/comments/{id}

index 2
baseURL/employees/{id}/messages
baseURL/employees/{id}/message/{id}

[martin_mueller]

Are you trying to group URLs while ignoring /{id} segments? If so, try this:

... | eval grouped_url = replace(httpRequestURL, "/\d+", "") | stats count by grouped_url

[jw44250]

but still splunk returns of URLS even i didnt ask for it...using case and searchmatch
it should return only that matching URL below why it returns other URLS which i did not ask for it
eval grouped_url=case(searchmatch("/api/v7/service/*"), "cloudsite.com",1=1,requestURI)

[jw44250]

i have 1000 of them with different paths

[jw44250]

grouped_url = /api/v1/appName/100/message/groups343434

How exclude things after id : grouped_url= /api/v1/appName/100

[jw44250]

i want to group by comments, employees,

Like Level 1, Level 2, Level 3

More Example

URI HTTP METHOD DESCRIPTION
/employee/add POST Add an employee
/employee/getDummy GET returns a dummy employee object
/employee/{id}/get GET Get the employee with ‘id’ in the URI
/employee/getAll GET Get all employees
/employee/{id}/delete DELETE Delete employee with ‘id’ in the URI

[jw44250]

Yes i want to ignore /{id} yes..

[niketn]

@jw44250... You would need to add some sample mock data and your existing metadata(field names)/existing query for us to come up with exact answer.. However, you can use the following in your case, assuming url is the field containing URLs in your log (if not you would need to perform Field Extraction using Splunk's Interactive Field Extraction or rex command):

<YourBaseSearchWithIndexSourceType>
| stats count(eval(match(url,"^baseURL/employees/\d+$"))) as EmployeeAccessCount count(eval(match(url,"^baseURL/employees/\d+/comments/\d+$"))) as CommentsAccessCount count(eval(match(url,"^baseURL/employees/\d+/messages/\d+$"))) as MessagesAccessCount

I have tested with following run anywhere search (you can change url and match condition as per your exact log data/field):

| makeresults
| eval url="baseURL/employees/12345/message/12345"
| table url
| stats count(eval(match(url,"^baseURL/employees/\d+/message/\d+$"))) as MessageAccessCount

[jw44250]

it does not working

[niketn]

@jw44250, your questions/requirements seems to be changing.
Since you have different types of URIs, I still expect that you should perform a match on URI with values like messages, comments, employees for you to come up with count etc. (you need to come up with cases based on your data):

eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
| table URI URIType

If you want to just remove digits and use the remaining URI, I would use martin_mueller's answer.

If you want to exclude things after ID like you have asked last, you can try: | eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")

Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")
| table URI requiredURI

On a different approach you can split URIs by forward slash character /. Then use mvindex to read what you are interested in. Like comments in the example below:

| eval arrURI=split(URI,"/")
| table URI arrURI
| search arrURI="comments"
| eval baseURI=mvindex(arrURI,1)
| eval commentNumURI=mvindex(arrURI,4)
| table URI arrURI baseURI commentNumURI

[jw44250]

thanks since im analyzing lots of logs file i mean 100s of them.
so getting urls like

URLS : - /1/ or /3/hello .

/20012/users

/%

[jw44250]

Excluding following urls using searchmatch

/xampp/zenario/
/xmb/
/xmldata
/xmlrpc.php
/xmlrpc/
/xmlrpcs.inc
/xoops/
/xoops/userinfo.php
/xoops_v2_rc3/html/
/xyzabc.jsp
/y4icpgtwvy0/
/yabb/
/yabbse-1.4.1/
/yabbse/
/yapig-0.92b/
/yapig-0.95b/
/yapig/
/yfPxl446.php4
/yfo267oib0m/
/ymGVHWTx.php
/z24d7fkor/
/zabbix/
/zapbook.cgi
/zapbook/
/zen-cart/
/zenario/
/zenariocms/
/zencart/
/200/
/1/
/2/
/600/anc
/zorum/
/zpanel/
/zsh

[niketn]

For your previous request for excluding list of urls using searchmatch, can you provide one full url for each as an example?

/xampp/zenario/
/xmb/
/xmldata
....
....

[jw44250]

How to replace localhost/employees/100/comments with digit using eval with searchMax

[niketn]

For your last question can you let me know what is the output you want when digits are replaced? The reason why I am asking is because martin_mueller has already given you a query to remove digits from URL. So please provide the expected output for the following:
input= localhost/employees/100/comments
output=?

[niketn]

@jw44250... You need to post URLs with code button ie. 10101010 so that it does not get omitted while posting. Please re-post URLs with the same.

PS: If my answers above do not cater to your needs, I can convert my answer to comment. However, the same will still not flag this question as unanswered question for all Splunker's. Splunk can definitely handle what you are trying to achieve also if @martin_mueller is answering a question you can be assured that it is the best solution 🙂

So may be simplify your question and ask what you really need.

[jw44250]

Since i have httpRequestURL as key in log files i am getting result i am looking for but i want group them in such away after main urls:

below example : matching employee with 100 and 800 are accessing comments url
localhost/employees/100/comments
localhost/employees/800/comments

matching 600 and 900 id having 3 messages
localhost/employees/600/messages/3
localhost/employees/900/messages/3

httpRequest
localhost/employees/100/comments
localhost/employees/200/comments/10
localhost/employees/300/messages
localhost/employees/400/message/3
localhost/employees/800/comments
localhost/employees/700/comments/10
localhost/employees/600/messages/3
baseURL/employees/400/message/3