topic Re: Filter partial duplicates in Splunk Search

How can I filter partial duplicates?

Yukie — Tue, 21 Feb 2023 15:24:42 GMT

Hello,

I'm new to splunk (Internship) and couldn't find and answer.

I'd need a way to filter my search.

I'm curently using a ".... | ... | stats count by RequestPath" search.

The problem is that the "RequestPath" can contain variable/random data at the end.

Exemple:
x/y/first

x/y/second/randomText

x/y/third

There are millions outputs and i would like to filter them so i only keep :

x/y/first

x/y/second

x/y/third

Thanks 🙂

Re: Filter partial duplicates

gcusello — Mon, 20 Feb 2023 11:28:43 GMT

Hi @Yukie,

you have to extract a part of the RequestPath using the rex command and use it for the stats command, something like this:

<your_search> | rex field=RequestPath "\w+\/\w+\/(?<SubPath>[^\/\n]+)" | stats count BY SubPath

Ciao.

Giuseppe

Re: Filter partial duplicates

Yukie — Mon, 20 Feb 2023 12:51:44 GMT

Hi @gcusello ,
Thanks for the fast answer.

It definitly helped but I realised it's a bit more complicated that what i described.

Your suggestion gives as output :

first
second
third

Where i'd like to have te full path until there
x/y/first

x/y/second

x/y/third

Because there might be for exemple :
x/y/first

x/z/second/random.pdf

x/y/third

Not an expert in regex and even less in splunk regex synthax. Sorry if it sound like something so simple i should have found myself.

Re: Filter partial duplicates

gcusello — Mon, 20 Feb 2023 13:01:39 GMT

Hi @Yukie,

no problem, please try this regex

^(?<SubPath>[^\/]+\/[^\/]+\/[^\/\n]+)

that you an test at https://regex101.com/r/0hzRax/1

Ciao.

Giuseppe