Splunk Search

How to get unique patterns

xvxt006
Contributor

Hi, we would like to get unique query string patterns so that we can cache them at Akamai. i have written a query
rex field=uri "\?(?.*)" | dedup command | table command

Below is the output. You can see 1,2,4,5 etc are same pattern. So how do i get just the unique patterns.

1 perPage=48&requestedPage=97
2 perPage=48&requestedPage=95
3 sst=subset
4 perPage=48&requestedPage=96
5 perPage=48&requestedPage=1
6 bc=y
7 redirect=socket+set
8 searchQuery=2LDR7&fromCatalog=true
9 ta=t
10 searchQuery=2LDR8&fromCatalog=true
11 searchQuery=cable+lug
12 searchQuery=stainless+pallet

Tags (4)
1 Solution

kristian_kolb
Ultra Champion

Hmm yes..I guess..maybe

your base search | dedup uri | rex field=uri mode=sed "s/=[^&]+/=XXX/g" | dedup uri | eval uri = substr(uri,2) | table uri

Haven't tested it, but it should replace all parameter values in the URL with 'XXX'. Yes there are 2 dedups. The first one is for reducing the number of uri's to be processed by rex. You might skip it if the uri's are almost always different.

This search does not take the order of the parameters into account, so

A=xxx&B=xxx
B=xxx&A=xxx 

would be considered different.

Should hopefully work...

/K

View solution in original post

0 Karma

emccaslin
Path Finder

Even better, use the 'cluster' command!


rex field=uri "?(?.*)" | cluster field=command | table command

With cluster you will get 2 fields, cluster_label (which is just the grouping number) and cluster_count (you many need to specify showcount=true, but this gives you the number of events in that grouping.

0 Karma

xvxt006
Contributor

Thank you. This view is also useful

0 Karma

kristian_kolb
Ultra Champion

Hmm yes..I guess..maybe

your base search | dedup uri | rex field=uri mode=sed "s/=[^&]+/=XXX/g" | dedup uri | eval uri = substr(uri,2) | table uri

Haven't tested it, but it should replace all parameter values in the URL with 'XXX'. Yes there are 2 dedups. The first one is for reducing the number of uri's to be processed by rex. You might skip it if the uri's are almost always different.

This search does not take the order of the parameters into account, so

A=xxx&B=xxx
B=xxx&A=xxx 

would be considered different.

Should hopefully work...

/K

0 Karma

xvxt006
Contributor

Thank you. So it will just remove the first character only?
so if i have uri as testuri?query=something, it will be esturi?query=something

0 Karma

xvxt006
Contributor

Even without eval uri = substr(uri,2) i am still getting the same results

0 Karma

xvxt006
Contributor

Hi K, one question - why do you have eval uri = substr(uri,2)?

0 Karma

kristian_kolb
Ultra Champion

This was some time ago, so I don't really remember 🙂

The effect is to remove the first character of the uri. If not needed - skip that step.

/k

0 Karma

xvxt006
Contributor

Hi Kristian why do you have substr(uri,2)?

0 Karma

xvxt006
Contributor

i define pattern as having different parameters (not worried about neither the order of the parameters as long as they are same parameters nor the values of them). So that is why i said 1,2,4,5 are one pattern (same parameters but different values. Even if same values also i am fine). number 3 is another pattern, number 6 and 7 are different patterns. 8 and 10 are same pattern, 11 and 12 are same pattern etc.

0 Karma

Ayn
Legend

How do you define "pattern"? Name and order of parameters?

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...