I have a set of related metrics I need to produce over a set of data
The initial part of the search looks something like
index=foo | bucket _time span=1h | stats count, median(duration), max(duration) by _time, blah
then I want to roll that up in different ways to produce the metrics. For example
1/ to get the peak count (and hour it occured)
| sort count desc | head 1
2/ to get the average median duration
| avg(median(duration))
etc
This works if I do separate full queries for each metric but is very inefficient as it repeats the main search each time
I would like to reuse the result set of the main search. I can sort of get this to work by using appendpipe
i.e.
index=foo | bucket _time span=1h | stats count, median(duration), max(duration) by _time, blah
| appendpipe [sort count desc | head 1]
| appendpipe [avg(median(duration))]
but of course it puts these rows at the end of the main result set. I can filter the results at the end to drop the rows from the initial search but this is ugly and inefficient.
And it breaks down when I introduce my next requirement - I need to repeat this process over different timespans. i.e. produce the same metrics rolled up for the last year, month and day. Given that the month is in the last year and so is the day, I wanted to have one years worth of results that I then segment off and crunch metrics off (e.g. | where _time >= relative_time(now(), @mon) and now crunch the metrics for the last month. Repeat for year and day.
Is there another way to do this? i.e. pass one result set through multiple pipes and aggregate the results?
... View more