I need help on doing cumulative percentiles, such as p90, over a period of time. This is different from rolling averages or taking the p90 of individual spans of time. For example, I'm trying to cal...
See more...
I need help on doing cumulative percentiles, such as p90, over a period of time. This is different from rolling averages or taking the p90 of individual spans of time. For example, I'm trying to calculate the cumulative, rolling p90 over a month. Here's a table that illustrates what I want: Day # P90 Value Calculation 2020-07-01 (Day 1) p90 of all events on day 1 2020-07-02 (Day 2) p90 of all events on day 1 and day 2 2020-07-03 (Day 3) p90 of all events from day 1 to day 3 ... ... 2020-07-31 (Day 31) p90 of all events during the month (day 1 - 31) If I wanted to do P90 for individual days, it's very simple: base_query | timechart span=1d p90(latency). Unfortunately, that's not what I want. There are 100,000+ events each day. I tried using streamstats but didn't get the expected results. I suspect that this is because streamstats only has a maximum window size of 10,000 events. My failed approach to using streamstats was to take the cumulative p90 of every event over a time period, and then return the final event of each day in that period: base_query | bin span=1d _time as day | streamstats p90(latency) as p90latency | dedup 1 day | sort by _time | table _time, p90latency Can anyone show me how to do a cumulative p90 in Splunk? I'm using an imperfect workaround of cumulative weighted average p90 values, but it's just an approximation and I'd like to have the real deal instead.