topic Why is Alert action not triggering when using real time? in Other Usage

Why is Alert action not triggering when using real time?

theprophet01 — Mon, 28 Aug 2023 18:53:21 GMT

-I am running an alert which is not triggering email actions when using real-time option. The alert is used to search for hosts which have not sent logs in the last 5 minutes.

-For example, I shut down a host for testing and wait 5 minutes. I then manually use the search string and specify time frame (e.g. last 15 minutes)- I am able to obtain results.

However, even though the same search was configured in the form of an alert running in real time, it produces no results nor does it trigger an email.

Here is the search I am using:

Re: Why is Alert action not triggering when using real time?

gcusello — Tue, 29 Aug 2023 06:10:03 GMT

Hi @theprophet01,

using a search like yours with Real-Time it isn't a good idea because you are using one CPU only for this search reducing the resources of your global Splunk infrastructure.

It's better to schedure a search e.g. every 5 minutes, so, when running is finished, the search releases the CPU for other jobs.

In addition, your search could be optimized to reduce the execution time and the CPU use:

| tstats max(_time) AS latest count BY host | eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") | where recent = 0 | rename host AS Host, realLatest AS "Latest Timestamp" | table Host, "Latest Timestamp"

At least, using this search you find only the hosts that didn't send logs in the last 5 minutes, but that sent logs in the previous 10 minutes (using a timeframe of 15 minutes); if your host doesn't send logs for 15 minutes you loose this information.

The best approach is having a lookup containing all the hosts to monitor (called e.g. perimeter.csv) containing at least one column (host) and running a search like the following:

in this way you have to manage the lookup but you have a more affidable control.

Ciao.

Giuseppe

Re: Why is Alert action not triggering when using real time?

theprophet01 — Tue, 29 Aug 2023 12:49:15 GMT

That worked perfect! I also used the 5 min scheduled search as suggested using a cron schedule. Thank you @gcusello you sir are indeed a legend!