How to run multiple macros in the same search?

[Raekmackbert]

Hi all!

I'm trying to run multiple macros in the same search and eventually aggregate the results from each execution into a table.

My current search looks like this, which seems to work fine for a single execution of the histperc macro (Prometheus integration provided)

| mstats rate(_value) AS requests WHERE "index"="MyIndex" AND metric_name="MyMetricNameRegex" BY metric_name, le
| stats sum(requests) AS total_requests BY metric_name, le
| `histperc(0.5, total_requests, le, metric_name)`
| rename histperc as Median
| table metric_name Median 90th 75th 25th 10th

I think the issue is that total_requests value is not passed down after the | `histperc(0.5, total_requests, le, metric_name)` row but i am not sure if this is the case. Also not sure if rename is by reference or copy and what would eventually happen by having many renames and overrides of the histperc value like below.

The histperc macro looks like this:

sort $groupby$, $le$
| eventstats max($hist_rate$) as total_hist_rate, last($le$) as uppermost_bound, count as num_buckets by $groupby$
| eval rank=exact($perc$)*total_hist_rate
| streamstats current=f last($le$) as gr, last($hist_rate$) as last_hist_rate by $groupby$
| eval gr=if(isnull(gr), 0, gr), last_hist_rate=if(isnull(last_hist_rate), 0, last_hist_rate)
| where $hist_rate$ >= rank
| dedup $groupby$
| eval res=case(lower(uppermost_bound) != "+inf" or num_buckets < 2, "NaN", lower($le$) == "+inf", gr, gr == 0 and $le$ <= 0, $le$, true(), exact(gr + ($le$-gr)*(rank - last_hist_rate) / ($hist_rate$ - last_hist_rate)))
| fields $groupby$, res
| rename res as "histperc"

What i want to do is something like this:

| mstats rate(_value) AS requests WHERE "index"="MyIndex" AND metric_name="MyMetricNameRegex" BY metric_name, le
| stats sum(requests) AS total_requests BY metric_name, le
| `histperc(0.5, total_requests, le, metric_name)`
| rename histperc as Median
| `histperc(0.9, total_requests, le, metric_name)`
| rename histperc as 90th
| `histperc(0.1, total_requests, le, metric_name)`
| rename histperc as 10th
| `histperc(0.75, total_requests, le, metric_name)`
| rename histperc as 75th
| `histperc(0.25, total_requests, le, metric_name)`
| rename histperc as 25th
| table metric_name Median 90th 75th 25th 10th

Thankful for all help!

[Raekmackbert · Accepted solution]

I found a solution that seems to work for me.. The issue was in the histperc macro when attempting to overwrite already created fields with new values. The query i came up with is

| mstats rate(_value) AS requests WHERE "index"="MyIndex" AND metric_name="MyMetricNameRegex" BY metric_name, le
| stats sum(requests) AS total_requests BY metric_name, le

| sort metric_name, le
| eventstats max(total_requests) as total_hist_rate, last(le) as uppermost_bound, count as num_buckets by metric_name
| eval rank=exact(0.5)*total_hist_rate
| streamstats current=f last(le) as gr, last(total_requests) as last_hist_rate by metric_name
| dedup metric_name

| `histperc_partial(Median, 0.5, total_requests, total_hist_rate, gr, last_hit_rate, num_buckets, uppermost_bound, le)`
| `histperc_partial(90th, 0.9, total_requests, total_hist_rate, gr, last_hit_rate, num_buckets, uppermost_bound, le)`
| `histperc_partial(75th, 0.75, total_requests, total_hist_rate, gr, last_hit_rate, num_buckets, uppermost_bound, le)`
| `histperc_partial(25th, 0.25, total_requests, total_hist_rate, gr, last_hit_rate, num_buckets, uppermost_bound, le)`
| `histperc_partial(10th, 0.10, total_requests, total_hist_rate, gr, last_hit_rate, num_buckets, uppermost_bound, le)`

| table metric_name Median 90th 75th 25th 10th

I had to create a new macro that does what the original histperc macro does but where fields are already calculated beforehand, the macro looks like

eval rank=exact($hist_perc$)*$total_hist_rate$
| eval $gr$=if(isnull($gr$), 0, $gr$), $last_hist_rate$=if(isnull($last_hist_rate$), 0, $last_hist_rate$)
| where $total_requests$ >= rank
| eval $fieldName$=case(lower($uppermost_bound$) != "+inf" or $num_buckets$ < 2, "NaN", lower($le$) == "+inf", $gr$, $gr$ == 0 and $le$ <= 0, $le$, true(), exact($gr$ + ($le$-$gr$)*(rank - $last_hist_rate$) / ($total_requests$ - $last_hist_rate$)))

Im sure there are ways to make this alot simpler and faster than what i have done, but this solved the issue for me at the moment. If anyone has ideas of where to improve i will gladly accept them, but for now i'll mark this question as answered

[smurf]

Hi,

I think the issue is that you are renaming the filed and using the original field again. When you rename it, the original field does not exist anymore, so in the next macro call you need to use the new field name.

[Raekmackbert]

Hi!

The histperc field is created by the execution of the macro, so wouldn't running the macro many times in sequence just recreate/update the field with a new value each time. 

I updated the original post with the actual macro as well in case you would like to see it (Even though the macro itself isnt mine)