Archive

How to group URLS based patterns?

New Member

I have n of log files and i'm getting the proper result for each URL as of now, but im facing issue since the same url can be access by the same employee not sure what is the best way i can do it.
Log files :

index 1
baseURL/employees
baseURL/employees/{id}

index 2
baseURL/employees/{id}/comments
baseURL/employees/{id}/comments/{id}

index 2
baseURL/employees/{id}/messages
baseURL/employees/{id}/message/{id}

Tags (4)
0 Karma

SplunkTrust
SplunkTrust

Are you trying to group URLs while ignoring /{id} segments? If so, try this:

... | eval grouped_url = replace(httpRequestURL, "/\d+", "") | stats count by grouped_url

New Member

but still splunk returns of URLS even i didnt ask for it...using case and searchmatch
it should return only that matching URL below why it returns other URLS which i did not ask for it
eval grouped_url=case(searchmatch("/api/v7/service/*"), "cloudsite.com",1=1,requestURI)

0 Karma

New Member

i have 1000 of them with different paths

0 Karma

New Member

grouped_url = /api/v1/appName/100/message/groups343434

How exclude things after id : grouped_url= /api/v1/appName/100

0 Karma

New Member

i want to group by comments, employees,

Like Level 1, Level 2, Level 3

More Example

URI HTTP METHOD DESCRIPTION
/employee/add POST Add an employee
/employee/getDummy GET returns a dummy employee object
/employee/{id}/get GET Get the employee with ‘id’ in the URI
/employee/getAll GET Get all employees
/employee/{id}/delete DELETE Delete employee with ‘id’ in the URI

0 Karma

New Member

Yes i want to ignore /{id} yes..

0 Karma

SplunkTrust
SplunkTrust

@jw44250... You would need to add some sample mock data and your existing metadata(field names)/existing query for us to come up with exact answer.. However, you can use the following in your case, assuming url is the field containing URLs in your log (if not you would need to perform Field Extraction using Splunk's Interactive Field Extraction or rex command):

<YourBaseSearchWithIndexSourceType>
| stats count(eval(match(url,"^baseURL/employees/\d+$"))) as EmployeeAccessCount count(eval(match(url,"^baseURL/employees/\d+/comments/\d+$"))) as CommentsAccessCount count(eval(match(url,"^baseURL/employees/\d+/messages/\d+$"))) as MessagesAccessCount

I have tested with following run anywhere search (you can change url and match condition as per your exact log data/field):

| makeresults
| eval url="baseURL/employees/12345/message/12345"
| table url
| stats count(eval(match(url,"^baseURL/employees/\d+/message/\d+$"))) as MessageAccessCount
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

New Member

it does not working

0 Karma

SplunkTrust
SplunkTrust

@jw44250, your questions/requirements seems to be changing.
Since you have different types of URIs, I still expect that you should perform a match on URI with values like messages, comments, employees for you to come up with count etc. (you need to come up with cases based on your data):

eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval URIType=case(match(URI,"comment") AND match(URI,"employee"),"EmployeeComment", match(URI,"message") AND match(URI,"employee"),"EmployeeMessage",1==1,"Employee")
| table URI URIType

If you want to just remove digits and use the remaining URI, I would use martin_mueller's answer.

If you want to exclude things after ID like you have asked last, you can try: | eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")

Following is run anywhere example:

| makeresults
| eval URI="localhost/employees/100/comments"
| eval requiredURI=replace(URI,"^([a-zA-Z]+\/[a-zA-Z]+\/\d+)\/([a-zA-Z]+)","\1")
| table URI requiredURI

On a different approach you can split URIs by forward slash character /. Then use mvindex to read what you are interested in. Like comments in the example below:

| eval arrURI=split(URI,"/")
| table URI arrURI
| search arrURI="comments"
| eval baseURI=mvindex(arrURI,1)
| eval commentNumURI=mvindex(arrURI,4)
| table URI arrURI baseURI commentNumURI
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

New Member

thanks since im analyzing lots of logs file i mean 100s of them.
so getting urls like

URLS : - /1/ or /3/hello .

/20012/users

/%

0 Karma

New Member

Excluding following urls using searchmatch

/xampp/zenario/
/xmb/
/xmldata
/xmlrpc.php
/xmlrpc/
/xmlrpcs.inc
/xoops/
/xoops/userinfo.php
/xoops_v2_rc3/html/
/xyzabc.jsp
/y4icpgtwvy0/
/yabb/
/yabbse-1.4.1/
/yabbse/
/yapig-0.92b/
/yapig-0.95b/
/yapig/
/yfPxl446.php4
/yfo267oib0m/
/ymGVHWTx.php
/z24d7fkor/
/zabbix/
/zapbook.cgi
/zapbook/
/zen-cart/
/zenario/
/zenariocms/
/zencart/
/200/
/1/
/2/
/600/anc
/zorum/
/zpanel/
/zsh

0 Karma

SplunkTrust
SplunkTrust

For your previous request for excluding list of urls using searchmatch, can you provide one full url for each as an example?

/xampp/zenario/
/xmb/
/xmldata
....
....

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

New Member

How to replace localhost/employees/100/comments with digit using eval with searchMax

0 Karma

SplunkTrust
SplunkTrust

For your last question can you let me know what is the output you want when digits are replaced? The reason why I am asking is because martin_mueller has already given you a query to remove digits from URL. So please provide the expected output for the following:
input= localhost/employees/100/comments
output=?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

SplunkTrust
SplunkTrust

@jw44250... You need to post URLs with code button ie. 10101010 so that it does not get omitted while posting. Please re-post URLs with the same.

PS: If my answers above do not cater to your needs, I can convert my answer to comment. However, the same will still not flag this question as unanswered question for all Splunker's. Splunk can definitely handle what you are trying to achieve also if @martin_mueller is answering a question you can be assured that it is the best solution 🙂

So may be simplify your question and ask what you really need.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

New Member

Since i have httpRequestURL as key in log files i am getting result i am looking for but i want group them in such away after main urls:

below example : matching employee with 100 and 800 are accessing comments url
localhost/employees/100/comments
localhost/employees/800/comments

matching 600 and 900 id having 3 messages
localhost/employees/600/messages/3
localhost/employees/900/messages/3

httpRequest
localhost/employees/100/comments
localhost/employees/200/comments/10
localhost/employees/300/messages
localhost/employees/400/message/3
localhost/employees/800/comments
localhost/employees/700/comments/10
localhost/employees/600/messages/3
baseURL/employees/400/message/3

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!