Deployment Architecture

Using outlier with grouping

caffein
Path Finder

How can I use outlier with grouping. For instance, if I want to group my data by country, I would like to remove outliers from each group's data, not from the population as a whole. This is a problem for me because some countries generate far more events than others, which skews the data. So, while a data point might seem like an outlier for the total population, it might be relatively normal for that particular country. Is there a way to do this?

Here's an example to help clarify:

user | country | foo | bar
---------------------------
1    | us      | 1   | 10
2    | us      | 2   | 12
3    | us      | 21  | 12
4    | ca      | 20  | 13
5    | ca      | 21  | 11

Ultimate output desired:
country | avg(foo) | avg(bar)
-----------------------------
us      | 1.5      | 11
ca      | 20.5     | 12

From the above, in the US users 3's foo value is an outlier, but that's a normal value for CA users. What I would like to be able to do is detect that user 3 is an outlier and discard that data, but keep the values for users 4 and 5 intact. Also, in my data there would be a lot more US events which would cause almost all the CA values to look like outliers.

Tags (1)
0 Karma
1 Solution

Ayn
Legend

A pretty similar question (as it seems to me at least) was posted a couple of days ago, and the question/answer/following discussion perhaps might help you get some inspiration on how to achieve your goal? http://splunk-base.splunk.com/answers/52107/how-do-i-remove-data-read-anomalies

View solution in original post

Ayn
Legend

A pretty similar question (as it seems to me at least) was posted a couple of days ago, and the question/answer/following discussion perhaps might help you get some inspiration on how to achieve your goal? http://splunk-base.splunk.com/answers/52107/how-do-i-remove-data-read-anomalies

caffein
Path Finder

Oh and just to put down how I think this would work:
[search] | eventstats median(foo) as medfoo, stdev(foo) as stdfoo by country | where abs(foo - medfoo)<stdfoo | table avg(foo) by country

0 Karma

caffein
Path Finder

I think that will do the trick. Thanks!

0 Karma

araitz
Splunk Employee
Splunk Employee

Oops, sorry, I just rememebered this issue. Let me think about this again.

0 Karma

caffein
Path Finder

Essentially I want to say
[search]...|outlier by country|table country, avg(foo), avg(bar)

I know outlier doesn't support "by", but that's basically what I'm going for.

0 Karma

caffein
Path Finder

Not really. My other question is about finding various box and whisker plot values for a full whole set of data. For this question I'm asking how I can loop through a set of groups and remove outliers within each group, rather than the population as a whole.

0 Karma

araitz
Splunk Employee
Splunk Employee

It seems like you have already solved this problem per your other question. Please correct me if I am mistaken.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...