I'm referencing this post - https://answers.splunk.com/answers/321226/how-to-create-an-alert-to-notify-me-via-email-when.html
I want to alert on pool violations, and gives me the correct info, but I'm not sure where to put the threshold check. I've put the "where" clause in a number of places, and each time, it returns nothing.
Also, is there a way to trigger a search when this happens and then send an email from the results?
The where clause goes at the end, but you gotta be careful with fields with spaces...i used single quotes to get it to work.
| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [ rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "% used"=round(used_bytes/quota*100,2) | fields Pool "% used" | where '% used' > .2
To send an email, hit Save AS > Alert and configure it run on a schedule and to trigger an email action when number of results is greater than 0 (and list in triggered alerts for verification/troubleshooting)
Also note @hexx answer that an alert for overall usage exists in the management console and can be triggered as an email alert as well.
Thanks, that worked for the evaluation. I'd like to be able to trigger a separate search through the alert - is that possible?
Yes. This alert will indicate a problem, and the customer is going to want some data to do research, so I want to then run a report giving him the needed data. Actually, something missing from the product - should be able to request an action that just points to another saved search and run it as an action. (At least, IMHO...)
A couple of options here:
yep, the alert action framework or the old run a script methods allow you to do what you want at that time. What do would you want to search if this alert hit?
@mmodestinosplunk, I've compared your search with the one in the DMC for the total license. This one seems to take into account you can have multiple license masters (splunkserver in the search). It also uses a join which is a bit more explicit than a "|search " in my opinion. So I've rewritten partly your search and the result is below.
However, in both cases, you consider a license group can only contain 1 single stack (by renaming stackids to stackid), is it really the case ?
Here's my search :
| rest splunk_server=local /services/licenser/pools | rename title AS pool | join type=outer splunk_server stack_id [ rest splunk_server=local /services/licenser/groups | eval stack_id=stack_ids | fields stack_id splunk_server is_active] | search is_active=1 | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "% used"=round(used_bytes/quota*100,2) | fields pool "% used" | where '% used' >= 0