Please find the below Sample Data.
In My above Data, there is a series of QueueNames (A, B, C, D, E) which will be indexed for every 5 mins into Splunk with Message_In_Queue value and Snapshot as time. There are RiskpointValue, ThresholdTime, MailID which are constant never change.
I need to generate the alert if Message_In_Queue Value is continuously greater then RiskpointValue in that ThresholdTime.
For Queue Name "A" Need to check the Last 10mins
If you see the QueueName A contains ThresholdTime as 10 mins and RiskpointValue is 1000. so for QueueName "A" we need to check the last 10 mins snapshots and compare the Message_In_Queue with RiskpointValue. if it is greater then in all the Snapshots we need to give an alert to the user(firstname.lastname@example.org).
For Queue Name "B" Need to check the Last 5mins
For Queue Name "C" Need to check the Last 15mins
For Queue Name "D" Need to check the Last 20mins
Please help me on how to achieve this dynamic case?
As you need to check the data for different time period, it is better to write alerts with schedule frequency same as ThresholdTime. So for your sample data you need to write 5 different alerts running every 5, 10, 15, 20 mins. Trigger the alert if following queries give output. Use $result.MailID$ in To field to send mail.
index = <index> QueueName = "A" earliest=-10m | where RiskpointValue > ThresholdTime | stats count by QueueName, MailID | where count >= 2 index = <index> QueueName = "B" earliest=-5m | where RiskpointValue > ThresholdTime index = <index> QueueName = "C" earliest=-15m | where RiskpointValue > ThresholdTime | stats count by QueueName, MailID | where count >= 3 index = <index> QueueName = "D" earliest=-20m | where RiskpointValue > ThresholdTime | stats count by QueueName, MailID | where count >= 4 index = <index> QueueName = "E" earliest=-5m | where RiskpointValue > ThresholdTime
Thanks Manjunath, for your response . Yes, I have already developed a query for individual alerts but what is the problem is there are 1000 Queues are in my case. So need to write 1000 separate alerts is difficult right?
So I am trying to achieve this in One Query or any Python Script?
Can you please help me further on how to achieve this?
You can group the QueueName values with same ThresholdTime. Configure Trigger Conditions to trigger alert for each result (alert.digest_mode = false).
index = <index> QueueName IN ("B", "E", "Z") earliest=-5m | where RiskpointValue > ThresholdTime
Do you have any idea to create an alert similar to UI on the click button?