topic Re: Calculate stats over a percentage of the data in Splunk Search

Calculate stats over a percentage of the data

gpburgett — Tue, 15 Feb 2011 12:37:37 GMT

I got a challenging request from a customer regarding their access logs. They want to monitor access patterns across all the the access points in their network by user. Particularly, they are interested in stats for the top 10% of users for each access point. (10%- the top ten out of 100 users). I've managed to get most of the info that they want using pretty simple searches, but I'm still stumped on this one:

total number of logins for the top 10% of users by access point

I've tried some things using subsearches and the perc() function, but the search string gets too complicated or I end up doing something that Splunk doesn't like. Maybe I'm overthinking it.

Here's my latest failure(the appendcols and where commands cause the problems):

sourcetype="access_log" |stats count as EVNT by APNUM USERID 
|appendcols [search sourcetype="access_log" | stats count AS EVNTCNT by APNUM USERID
             | stats p90(EVNTCNT) as LIM by APIP| fields APIP LIM ]
| eval USE=if(EVNT<LIM, "NO", "YES")| table APIP, EVNT, LIM, USE| stats sum(EVNT) by APIP| where USE=YES

Re: Calculate stats over a percentage of the data

gpburgett — Tue, 15 Feb 2011 14:21:04 GMT

It looks like the "where" command can only be used after a search string and not after a function. Correct?

Re: Calculate stats over a percentage of the data

gpburgett — Mon, 28 Sep 2020 09:24:44 GMT

The search string got cut off. Here's the complete search:

Re: Calculate stats over a percentage of the data

jrodman — Thu, 17 Feb 2011 03:06:45 GMT

Just tried to hack up the searchstring to be a bit more readable in answers

Re: Calculate stats over a percentage of the data

jrodman — Thu, 17 Feb 2011 03:10:24 GMT

Are logins and accesses the same?
Are we starting with the set of logins, and wanting to find, for each access point, the top 10% of users and their count of logins?

Re: Calculate stats over a percentage of the data

sideview — Thu, 17 Feb 2011 03:45:45 GMT

One note is that "| where USE=YES" is going to look for rows where the value of the USE field is equal to the value of the YES field. If you mean the literal 3-character value YES, you have to put the YES in quotes. Where is a little different from search and that's one of the ways.

Re: Calculate stats over a percentage of the data

gpburgett — Thu, 17 Feb 2011 08:20:01 GMT

Thanks for the formatting help!

Yes, I mean the same thing when I say "access" and "login". I have a set of access logs and I want to find the total count of accesses for the top 10% of users per access point.

Re: Calculate stats over a percentage of the data

steveyz — Thu, 17 Feb 2011 14:03:08 GMT

sourcetype=wims_auth | stats count as EVNT by APIP MACID | eventstats perc90(EVNT) as cutoff by APIP | where EVNT>=cutoff | stats sum(EVNT) by APIP