How to pass a macro'ed value (string or number) to another macro?

[rjthibod]

I currently use various macros to store default values (thresholds, static filter strings, etc.) in an app. These default value macros (especially the numerical ones) can be changed by a customer/organization to suit their use cases/environment.

I have other search macros that use those default values in their pipepline. For example, assume there is a default alert threshold for a metric, and I have a search macro that use that threshold to trigger alerts, write to a summary index, etc.

My question is, what is the appropriate way to pass those default threshold/alerting macro values into search-related macros? I have devised a solution that I will share, but I wanted to see if anyone else has a suggestion.

Note: assume lookups are not really an option. Besides, I am pretty sure my solution would work the same for lookups.

Here are is more detail about an example set of macros:

[default_metric_5min_threshold]
description = Default threshold (in seconds) to trigger a alert for my custom sourcetype
iseval = 0
definition = 100

...

[alerts_default_metric_search_5min_span]
description = Search for calculating alert using default alert threshold value. Assume 5-min sliding window to aggregate data.
iseval      = 0
definition  = <INSERT_MACRO_MAGIC>

[sideview · Accepted solution]

Apologies in advance, but I'm going to reframe the problem and restate broader facts about nested macros and args etc. I think this is either a really complex question, or on the other extreme something really weird is happening that's making macros misbehave on your system.

1) You can of course use macros within the definition of other macros, like this:

[default_bins]
description = default bins for various timecharts.
definition = 400

[timechart_macro]
definition = timechart bins=`default_bins` count

Which then an end-user would use like so:

<your search> | `timechart_macro`

2) You can even have it both ways, where the end-user can either a) pass an explicit argument, or b) omit it and have some macro-configurable default apply.

[timechart_macro(1)]
args = bins
definition = timechart bins=$bins$ count

At which point your users are free to use either:

<your search> | `timechart_macro`

or

<your search> | `timechart_macro(500)`

Which is cool, but pretty common in the field. I'm not saying anything new (or answering your question yet!!).

So returning to your question....the best I can do is ask a question back -- why doesn't the below solution work for you? It seems that for some reason it doesn't, but I think that reason is the very thing that I'm missing about the question. My apologies also if the facepalm here is mine.

# Note that you need a (1) after the macro name in the stanza
[alerts_default_metric_search_5min_span(1)]
description = gets run when the end-user specifies an explicit threshold, like `alerts_default_metric_search_5min_span(25)`
args = threshold
definition  = tstats .... | where metric >= $threshold$ | <do other actions>

[alerts_default_metric_search_5min_span]
description = This is instead run when the end-user runs the macro with no explicit threshold, like `alerts_default_metric_search_5min_span`
definition  = tstats .... | where metric >= `default_metric_5min_threshold` | <do other actions>

[rjthibod]

@sideview

Thanks as always for catching errors and being so detailed. The missing "(1)" from the stanza was just an oversight when typing and copying various things.

Your points about the various things you can do are correct and I use them heavily in my macro definitions. To carry the point home even more, everything I am doing and suggesting in my question and first answer work fine; I am simply asking if someone thinks they now a better way to do what I am doing.

To your last suggestion about using the default value macro in the search macro, you are spot on that I could do that. I don't use that approach in my definitions, because I wanted to write only a single macro that actually defines the full search (many, many pipes long). In this case the macro alerts_default_metric_search_5min_span(1) is the only place where I define the full search, and the app can pass the default threshold value as I do in my example, or another user could come in and pass their own value. With my way, it is the same macro with a flexible argument instead of having to define and maintain two macros that are exactly the same less that one takes a user-entered value for the threshold and the other takes no arguments and hard codes the default threshold macro in the search.

Now, none of the above may make sense so let me rephrase in the most generic way possible.

I have a complicated search macro "alerts_default_metric_search_5min_span" that takes a single argument. I don't want multiple copies of the full search macro, so I had to devise a way to pass the macro'ed default threshold value into that search, or allow a user to pass their own threshold value. Because you cannot pass a macro into another macro like this alerts_default_metric_search_5min_span(alerts_default_metric_search_5min_span), I devised the trick with gentimes and map.

The question is, is there some other way I am not considering to accomplish the same thing without making a copy of the really nasty search macro?

[sideview]

Oh I see. Well it may feel a bit clunky, but if you don't want to repeat a lot of complex SPL, you can clip it off into a pair of macros, like so.

[default_metric_5min_threshold]
description = Default threshold
definition = 100

[alerts_default_metric_search_5min_span(1)]
args = threshold
definition  = `macroA` | where metric >= $threshold$ | `macroB`

[alerts_default_metric_search_5min_span]
definition  = `macroA` | where metric >= `default_metric_5min_threshold` | `macroB`

[macroA]
definition = | tstats <complex tstats args> | <other complex SPL>

[macroB] 
definition = <other complex SPL>

And I think that sort of approach is a bit better than anything that relies on the map command. For one thing I think the map command will force all those rows to come back to the search head if something wasn't already, ie instead of running that SPL out at the indexers.

[rjthibod]

Point taken. I will do some poking around with the job inspector to see if I can detect a difference with and without using map.

The performance of the potentially most concerning instance of my approach has seemed to perform well, but I cannot be for certain. It is a convoluted search tstats based search against datamodels, so I need to dig into the inspector.

Regarding map and its semantics, is that documented anywhere beyond the standard Splunk SPL page? I am looking for material where it talks about the impact caused by bringing data to the search heard versus the indexer, not the calling semantics (I got that part down).

[sideview]

No I don't know of anywhere that this is documented. I know that the streamstats command, even though it's technically 'streaming non-transforming', will force the rows to be brought back to the search head (so that it itself will run there). The reason being that streamstats needs to make an assumption that it's marching through the overall resultset in order, and that requires that the rows be merged. I believe and assume that the map command must make essentially the same assumption and as such the work can't be pushed out to the indexers. That said, depending on the context of our search, other commands may have forced the rows to the Search head already by that point.

[rjthibod]

Thanks for your help as always. Forgot to accept this as an acceptable answer.

[rjthibod]

Here is my own solution. Please let me know if you think it is a good, advisable one or not.

The trick I came up with (since macro arguments cannot contain other macros) is to use gentimes and map to pass the macro'ed values into the other macros.

For example, this is the updated definition of the no arguments macro in my post "alerts_default_metric_search_5min_span"

definition = gentimes start=-1 | eval thresh = `default_metric_5min_threshold` | table thresh | map search="`alerts_default_metric_search_5min_span($thresh$)`"

And then I have another definition for a one argument version of "alerts_default_metric_search_5min_span" that actually performs the search,

 [alerts_default_metric_search_5min_span(1)]
 description = Search for calculating alert using default alert threshold value. Assume 5-min sliding window to aggregate data.
 args = threshold
 iseval      = 0
 definition  = tstats .... | where metric >= $threshold$ | <do other actions>

[rmarcum]

Could you clarify "since macro arguments cannot contain other macros". I assume this is not meaning nested macros, which work well and I use in many solutions. Also, this question site contains many such explanations for this. Regards.

[rjthibod]

My statement is specific to the limitation on using macros in the argument list of a macro (e.g., passing a macro in an argument to another macro). That is not allowed.

I too make liberal use of macros inside of the definitions (expansion) of other macros.