topic Defining a global token for alert recipients in Splunk Enterprise

Defining a global token for alert recipients

fatsug — Wed, 24 Jul 2024 17:00:50 GMT

I have been trying to achieve "grouped email recipients" and while it is possible, it just won't behave the way I want with generative commands.

For "raw events" it works great to have a macro with an eval setting "recipients" to a list of email adresses and then using $result.recipients$ in the "action.email.to ="

Howerver, for things like stats and table, this does not work as the actual values of recipients are not part of the results.

So for "table" it works if I include "recipients" in the table, but that looks horrible. This can be sort of demonstrated like so where this works:

index="_internal" | `recipients` | dedup log_level | table log_level | fields recipients

And this does not

index="_internal" | eval recipients = "email1@email.com, email2@email.com" | dedup log_level | table log_level | fields recipients

As recipients is empty

So, someone suggested that one could use a savedsearches.conf.spec file to define a token like:

[savedsearches] recipients = <string>

and then use "recipients" in the savedsearches.conf file as $recipients$. This does not seem to be the case though, I cannot find this documented anywhere and the spec file seems to be more "instructive" than anything.

Another suggestion was to define global token directly in the savedsearhes file like:

[tokens] recipients = Comma-separated list of email addresses

and then use $recipients$ for all "action.email.to = $recipients$" in that file. Though I cannot find the token definition solution here documented anywhere.

Are any of these suggestions at all valid? Is there any way to somewhere in the app where the alerts live to define a "token" like "recipients" which can be referenced in all "action.email.to" instances in that file so that I only have to update one list in one place?

Or is this a "suggested improvement" I need to submit somewhere 🙂

All the best

Re: Defining a global token for alert recipients

marnall — Wed, 24 Jul 2024 19:12:43 GMT

You can set recipients as a hidden field by prepending '_' to the field name. This will prevent the recipients column from appearing in the table, but the token will still work.

| eval _recipients = "email1@email.com, email2@email.com"

Then use:

$result._recipients$ in the "action.email.to ="

I would also suggest putting this _recipients eval at the end of your search so it does not accidentally get removed by things like "table". It should also work if you put the eval statement into a macro.

Re: Defining a global token for alert recipients

fatsug — Thu, 25 Jul 2024 08:27:51 GMT

Sweet relief after so much trial and error, I could kiss you!

Yes, this solution finally works!

savedsearches.conf

<basesearch> | table <something> | `macro`

macro.conf

[macro] definition = eval _recipients="email1@email.com, email2@email.com"

and finally in the savedsearches.conf (or To: field in the UI)

action.email.to = $result._recipients$

And it finnaly works as intended!!! 👏 Whish I could reward 100 karma for this 😄

Still think that this should be a "built in" thing available both in the GUI and config files, "email groups", but I'm to happy to care right now

Re: Defining a global token for alert recipients

fatsug — Tue, 30 Jul 2024 12:32:36 GMT

This solution still works for most cases, however if you need an alert where number of events is 0 then this solution will not work, not "as is" at least.

As a search for, lets say a problem with logshipping, should alert on 0 returned events. Then there is no way to hitch on a hidden field to anything as there are no results.

So to fill "all my needs" here, I would have to come up with something completely different, it would need to become a feature in Splunk or I have to sort out an manage a number of group recipients in exchange.

Re: Defining a global token for alert recipients

marnall — Wed, 31 Jul 2024 19:01:39 GMT

You can use the following segment to make the alert trigger even when its search returns zero events:

<yoursearch> | appendpipe [ stats count | eval description="No problems found. All is well!" | where count = 0 | fields - count]

If there are results from the initial search, then this segment does not change the results. But if there are no results from the initial search, this segment will create a single row with a single field of "description" containing the string.

Re: Defining a global token for alert recipients

fatsug — Wed, 31 Jul 2024 21:38:15 GMT

Sweet

I re-wrote the "if 0 events" to "if there were events but now there are none"

This way I suppose the alert should be set to if there are "1 events"? If the initial search returns 0 events then there will be 1 generated event with the field "description".

Thanx

Re: Defining a global token for alert recipients

marnall — Thu, 01 Aug 2024 18:18:56 GMT

Yes, if you add the segment in my last reply then your alert will always find at least one result.

Re: Defining a global token for alert recipients

fatsug — Fri, 02 Aug 2024 06:37:21 GMT

You know, sometimes you know something but until you really test all options you're just not sure.

WIth a "count", like

| tstats count WHERE index="<index>" earliest="-5min" latest=now()
| `<mail_macro>` 
| rename count as "Events"

There will always be at least one result "0" (zero). It also does not matter if the count is 0 or 99999999, there is exactly 1 result. So the email macro does work, the condition "Number of results = 0" just fails, and it will fail producing false positives with ">=1" as well.

I forgot about "custom trigger conditions" though, which is likely the best solution for the intended usecase.

| tstats count WHERE index="<index>" earliest="-5min" latest=now()
| eval Information = if(count="0", "Currently f-d","Working")
| `<mail_macro>` 
| rename count as "Events"

Then using a "custom trigger" like 'search Events = "Currently f-d"' works just as well as the solution outputing only results where there were events last x minutes but are zero events current x minutes.

Probably more effective as well.

Thank you both for your help, the community here is fantastic

Re: Defining a global token for alert recipients

fatsug — Fri, 02 May 2025 17:48:09 GMT

"Email recipients groups" | Ideas