I would like to monitor each individual queue. The alarm can be parameterized for individual queues via lookup.
My goal is to build a dashboard in a way that each line does not only show the monitored queue but also the method (see below 3 methods) with which this queue should be monitored. How can I pack the results triggered by notifications into a dashboard using a lookup table?
The method consists of 3 notifications for each individual queue:
1) Notification for missing message flow
After the alarm is triggered, how long will no messages flow in minutes?
2) High latency notification [duration=delivery-entry]
(e.g. if duration > 1200
)
What is the latency in seconds until an alarm is triggered?
3) Notification of high queue level (e.g. queue_level > 100
)
What is the level in number of messages, from which an alarm is triggered?
-The time until the next same alarm is triggered may be specified in minutes.
(1) Example of a queue that has not generated a message flow in the last 20 minutes:
1.1 Search:
index=hogehoge earliest=-20m@s sourcetype=syslog queue="system1"
1.2 Alerts:
Number of results is zero. Planned. Cron schedule
Also, I.e. for this queue (queue="system1" )
I should create 3 alerts (because 3 methods should be monitored for this queue) and put the results of alerts into the dashboard to make it clear. So far I haven't found a solution how to automatically create alerts for multiple queues (queue="system2" OR queue="system3" ...)
and pack their results into the lookup table.
I would be very grateful for their support.
Thanks
If you want to push the results of an Alert into a lookup table, you merely need to put a line like the following in:
| outputlookup [append=true] <lookupname>
See https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/outputlookup for more details