Is it possible to create Alerts for multiple Instances of an Application?

[ninivmat]

I'm currently trying to set up alerts if an instance of our application is down.
However we have 40 Instances and I want to create an individual alert for each of them.

My log event is something of the kind of :

[timestamp] instance_name=test1 instance_state=0 description="lorem ipsum foo bar"
[timestamp] instance_name=test1 instance_state=1 description="lorem ipsum foo bar"
[timestamp] instance_name=test2 instance_state=0 description="lorem ipsum foo bar"
[timestamp] instance_name=test3 instance_state=0 description="lorem ipsum foo bar"

I created the search to find all instances that are down in a given timespan and know a little bit on how to set up alerts. However, when I set up my alert it always triggers as soon as any instance is down.

I want a separate alert if test1 is down and another if test2 is down, or elsewise I will get flooded with mails because we have a lot of instances. Also different users want to be informed for different instances (not all of them).

Is there a way to do this clean and proper or do I have to set up a search for each instance and create an alert for each instance? Is there a way to set up the aler to something regex-ish like test[0-9] and create individual mails for each match?

My current option would be to create about 50 individual alerts, which seems to me like a quick'n'dirty inproper approach.

Any links, tipps even search querys would help, since I'm not even sure what to search for on google and here....

[kamlesh_vaghela]

Hi ninivmat,

Here you have to just create single alert which monitors all instances and trigger a mail for down instances.

First, create a search which lists only down instances. Create Alert with this search. Which will execute in every 5 minutes and mail to admin for each instance which is down. It may possible it will flood with alerts emails. So we can Throttle alerts.

Here I've created sample alert for you. Can you please put it into of search app??

[Alert - Down Instances]
action.email = 1
action.email.inline = 1
action.email.sendresults = 1
action.email.to = test@test.com
action.email.useNSSubject = 1
alert.digest_mode = 0
alert.suppress = 1
alert.suppress.fields = instance_name
alert.suppress.period = 15m
alert.track = 0
counttype = number of events
cron_schedule = 0-59/5 * * * *
description = Alert - Down Instances
enableSched = 1
quantity = 0
relation = greater than
search = index=Your_INDEX | dedup instance_name  | search instance_state=0 | table instance_name

Change action.email.to and index=Your_INDEX in search.

Please refer below links.

http://docs.splunk.com/Documentation/Splunk/7.0.0/Alert/Setupalertactions
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Alert/Alertexamples
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Alert/ThrottleAlerts

Let me know for further assistance.

Thanks

[ninivmat]

But this would generate an alert if any instance is down and give me a list of the instances that are down.

My use case is that we have different teams for testing on multiple instances (40) and they want to be notified when their instance is down, but don't care about the instances of other teams....

So i want to generate different mails with alerts, based on the instance_name...
Do I need to do this with multiple alerts or can it be done elsewise

[kamlesh_vaghela]

Hi

Is it possible to store email_id and instance name in any csv (lookup file) ? If yes then we can send mails to respective users from the single Alert.

We have to just add lookups into search and use email_id from the results.
like.

action.email.to = $result.email_id$

So it will send mail to respective users.

Thanks