https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50206#M22207 <P>How can I use outlier with grouping. For instance, if I want to group my data by country, I would like to remove outliers from each group's data, not from the population as a whole. This is a problem for me because some countries generate far more events than others, which skews the data. So, while a data point might seem like an outlier for the total population, it might be relatively normal for that particular country. Is there a way to do this?</P> <P>Here's an example to help clarify:</P> <PRE><CODE>user | country | foo | bar --------------------------- 1 | us | 1 | 10 2 | us | 2 | 12 3 | us | 21 | 12 4 | ca | 20 | 13 5 | ca | 21 | 11 Ultimate output desired: country | avg(foo) | avg(bar) ----------------------------- us | 1.5 | 11 ca | 20.5 | 12 </CODE></PRE> <P>From the above, in the US users 3's foo value is an outlier, but that's a normal value for CA users. What I would like to be able to do is detect that user 3 is an outlier and discard that data, but keep the values for users 4 and 5 intact. Also, in my data there would be a lot more US events which would cause almost all the CA values to look like outliers.</P> Fri, 11 May 2012 18:06:21 GMT caffein 2012-05-11T18:06:21Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50206#M22207 <P>How can I use outlier with grouping. For instance, if I want to group my data by country, I would like to remove outliers from each group's data, not from the population as a whole. This is a problem for me because some countries generate far more events than others, which skews the data. So, while a data point might seem like an outlier for the total population, it might be relatively normal for that particular country. Is there a way to do this?</P> <P>Here's an example to help clarify:</P> <PRE><CODE>user | country | foo | bar --------------------------- 1 | us | 1 | 10 2 | us | 2 | 12 3 | us | 21 | 12 4 | ca | 20 | 13 5 | ca | 21 | 11 Ultimate output desired: country | avg(foo) | avg(bar) ----------------------------- us | 1.5 | 11 ca | 20.5 | 12 </CODE></PRE> <P>From the above, in the US users 3's foo value is an outlier, but that's a normal value for CA users. What I would like to be able to do is detect that user 3 is an outlier and discard that data, but keep the values for users 4 and 5 intact. Also, in my data there would be a lot more US events which would cause almost all the CA values to look like outliers.</P> Fri, 11 May 2012 18:06:21 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50206#M22207 caffein 2012-05-11T18:06:21Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50207#M22208 <P>It seems like you have already solved this problem per your other question. Please correct me if I am mistaken.</P> Thu, 17 May 2012 22:07:38 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50207#M22208 araitz 2012-05-17T22:07:38Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50208#M22209 <P>Not really. My other question is about finding various box and whisker plot values for a full whole set of data. For this question I'm asking how I can loop through a set of groups and remove outliers within each group, rather than the population as a whole.</P> Thu, 17 May 2012 22:39:21 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50208#M22209 caffein 2012-05-17T22:39:21Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50209#M22210 <P>Essentially I want to say<BR /> [search]...|outlier by country|table country, avg(foo), avg(bar)</P> <P>I know outlier doesn't support "by", but that's basically what I'm going for.</P> Fri, 18 May 2012 00:57:02 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50209#M22210 caffein 2012-05-18T00:57:02Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50210#M22211 <P>Oops, sorry, I just rememebered this issue. Let me think about this again.</P> Fri, 06 Jul 2012 18:16:51 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50210#M22211 araitz 2012-07-06T18:16:51Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50211#M22212 <P>A pretty similar question (as it seems to me at least) was posted a couple of days ago, and the question/answer/following discussion perhaps might help you get some inspiration on how to achieve your goal? <A href="http://splunk-base.splunk.com/answers/52107/how-do-i-remove-data-read-anomalies">http://splunk-base.splunk.com/answers/52107/how-do-i-remove-data-read-anomalies</A></P> Fri, 06 Jul 2012 18:32:11 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50211#M22212 Ayn 2012-07-06T18:32:11Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50212#M22213 <P>I think that will do the trick. Thanks!</P> Fri, 06 Jul 2012 18:55:34 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50212#M22213 caffein 2012-07-06T18:55:34Z https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50213#M22214 <P>Oh and just to put down how I think this would work:<BR /> [search] | eventstats median(foo) as medfoo, stdev(foo) as stdfoo by country | where abs(foo - medfoo)&lt;stdfoo | table avg(foo) by country</P> Fri, 06 Jul 2012 20:10:24 GMT https://community.splunk.com/t5/Deployment-Architecture/Using-outlier-with-grouping/m-p/50213#M22214 caffein 2012-07-06T20:10:24Z