Splunk Search

How to alert on license pool violation and can i trigger a search upon the violation

Champion

Hi,

I'm referencing this post - https://answers.splunk.com/answers/321226/how-to-create-an-alert-to-notify-me-via-email-when.html

I want to alert on pool violations, and gives me the correct info, but I'm not sure where to put the threshold check. I've put the "where" clause in a number of places, and each time, it returns nothing.

Also, is there a way to trigger a search when this happens and then send an email from the results?

0 Karma

Contributor

@mmodestinosplunk, I've compared your search with the one in the DMC for the total license. This one seems to take into account you can have multiple license masters (splunkserver in the search). It also uses a join which is a bit more explicit than a "|search []" in my opinion. So I've rewritten partly your search and the result is below.

However, in both cases, you consider a license group can only contain 1 single stack (by renaming stackids to stackid), is it really the case ?

Here's my search :

| rest splunk_server=local /services/licenser/pools 
| rename title AS pool 
| join type=outer splunk_server stack_id 
 [ rest splunk_server=local /services/licenser/groups 
 | eval stack_id=stack_ids
 | fields stack_id splunk_server is_active]
| search is_active=1
| eval quota=if(isnull(effective_quota),quota,effective_quota) 
| eval "% used"=round(used_bytes/quota*100,2) 
| fields pool "% used"
| where '% used' >= 0
0 Karma

Splunk Employee
Splunk Employee

Hi a212830!

The where clause goes at the end, but you gotta be careful with fields with spaces...i used single quotes to get it to work.

| rest splunk_server=local /services/licenser/pools 
| rename title AS Pool 
| search 
    [ rest splunk_server=local /services/licenser/groups 
    | search is_active=1 
    | eval stack_id=stack_ids 
    | fields stack_id] 
| eval quota=if(isnull(effective_quota),quota,effective_quota) 
| eval "% used"=round(used_bytes/quota*100,2) 
| fields Pool "% used"
| where '% used' > .2

alt text

To send an email, hit Save AS > Alert and configure it run on a schedule and to trigger an email action when number of results is greater than 0 (and list in triggered alerts for verification/troubleshooting)

alt text

Also note @hexx answer that an alert for overall usage exists in the management console and can be triggered as an email alert as well.

Champion

Thanks, that worked for the evaluation. I'd like to be able to trigger a separate search through the alert - is that possible?

0 Karma

Splunk Employee
Splunk Employee

yep, the alert action framework or the old run a script methods allow you to do what you want at that time. What do would you want to search if this alert hit?

0 Karma

Ultra Champion

As in, make the search trigger another search?

0 Karma

Champion

Yes. This alert will indicate a problem, and the customer is going to want some data to do research, so I want to then run a report giving him the needed data. Actually, something missing from the product - should be able to request an action that just points to another saved search and run it as an action. (At least, IMHO...)

0 Karma

Ultra Champion

A couple of options here:

  • Have you validated that no such thing already exists on splunkbase? The new mod-alert framework is designed for such extensibility
  • Pre mod-alert, we used to solve this by include a dynamic URL field in the results. That way when the alert is triggered, one of the fields that would be sent to the customer (assuming you include the results in the alert) is the URL that will direct them to learn more
0 Karma