How to implement alert that need to consider state of past alert?

[yshen]

I need to first issue an alert for overheat temperature 24 hours in advance for the affected locations, for their forecast to be above 100 F (long term query).

Then I need to query for the next 2 hours to 8 hours (for near term forecast), of the more recent temperature forecast for the same sets of locations.

If the recent forecast for the same location has dropped below the threshed 100 F, I need to issue an alert to cancel the previous alert.

If a location's recent forecast is above 100 F, but the prior forecast was below 100 F (no alert had been issued), I need to issue a new alert for the location.

Effectively, the query for near term forecast needs to access the query results of the long term query (or redo a query for the previous long term query), to compare with the recent forecast results.

(I'm especially not clear how to compare two queries' results with Splunk query.) I wonder how to implement a solution with Splunk? Thanks for pointers!

Let's build an example to develop the solution.

Assume the operation time in question is 8:00 AM on July 14, 2022,
so the 24 hour in advance long term forecast should have been made at 8:00 AM on July 13, 2022 (long term forecast) 

The time window to make the short term forecast should be 0:00 AM (8-8) and 6:00 AM (8-2) (8 to 2 hours before) on the same day. 

Here is more concise requirements:

1.	Hourly, the forecasts of 24 hours after for all locations shall be collected and evaluated. If the 24-hour-after temperature will be over the threshold (100 F), alert shall be sent for the to-be-overheat locations.
2.	Also hourly, the forecasts for the window of next 2 hours to the next 8 hours should be collected and evaluated. Based on the evaluation of the 2-hours-8-hours-after forecast, revision shall be made according to the following rules:
a.	If a location’s 2-hours-8-hours-after forecast is below the threshold, while there had been an alert issued. A cancellation message shall be sent.
b.	If a location’s 2-hours-8-hours-after forecast is above the threshold, while there had not been alert sent, then a new alert shall be sent
c.	For the other case, no operation is needed
3.	At 15 minutes interval, the real time temperature for the locations shall be collected and evaluated. . Based on the evaluation of the real-time temperature, revision shall be made according to the following rules:
a.	If a location’s real time temperature is below the threshold, while there had been an alert issued. A cancellation message shall be sent.
b.	If a location’s real time temperature is above the threshold, while there had not been alert sent, then a new alert shall be sent
c.	For the other case, no operation is needed

[gcusello]

Hi @yshen,

the only way to do something near your need is that your alert writes a row in a lookup or in summary index, then you have a second alert that checks the condition of the previous fired alerts on the lookup (or summary index).

You can do this using "outputlookup" command (for lookups) or "collect" command (for summary indexes).

Ciao.

Giuseppe

[yshen]

@gcusello Thanks for the pointers. They are indeed helpful.

Lookup table or summary index, which is more appropriate?

I'm not yet fluent in Splunk query programming. I wonder if Splunk custom alert action script would be an easier alternative to implement the evaluation logic?

Any example of similar query programming, or of custom alert action script would be appreciated!

Thanks again!

[gcusello]

Hi @yshen,

if you're not fluent in SPL I hint to follow the Splunk Search Tutorial (https://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial) because searches are the base of everything in Splunk!

Anyway the choose between lookup and summary index depends on some factors:

if you have many cases Summary is more efficient, if for the check the time isn't relevant, lookup is the easier way.

Anyway, you have to take your alert and add at the end the command

| outputlookup your_alerts_Lookup.csv append=true

then when you run an alert you can search on the lookup.

It's difficoult to give you more help because it depends on the use cases.

For this reason I hinted to learn SPL, so develop this solution will be very easy.

Ciao.

Giuseppe