Why is Alert action not triggering when using real time?

[theprophet01]

-I am running an alert which is not triggering email actions when using real-time option.   The alert is used to  search for hosts which have not sent logs in the last 5 minutes.

-For example, I shut down a host for testing and wait 5 minutes. I then manually use the search string and specify time frame (e.g. last 15 minutes)- I am able to obtain results.

However,  even though the same search was configured in the form of an alert running in real time, it produces no results nor does it trigger an email.

Here is the search I am using:

index=* | stats max(_time) as latest by host | eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") | fields - latest | where recent = 0 | rename host AS Host, realLatest AS "Latest Timestamp" | table Host, "Latest Timestamp"

[theprophet01]

That worked perfect! I also used the 5 min scheduled search as suggested using a cron schedule. Thank you @gcusello you sir are indeed a legend!

[gcusello · Accepted solution]

Hi @theprophet01,

using a search like yours with Real-Time it isn't a good idea because you are using one CPU only for this search reducing the resources of your global Splunk infrastructure.

It's better to schedure a search e.g. every 5 minutes, so, when running is finished, the search releases the CPU for other jobs.

In addition, your search could be optimized to reduce the execution time and the CPU use:

| tstats max(_time) AS latest count BY host
| eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") 
| where recent = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

At least, using this search you find only the hosts that didn't send logs in the last 5 minutes, but that sent logs in the previous 10 minutes (using a timeframe of 15 minutes); if your host doesn't send logs for 15 minutes you loose this information.

The best approach is having a lookup containing all the hosts to monitor (called e.g. perimeter.csv) containing at least one column (host) and running a search like the following:

| tstats max(_time) AS latest count BY host
| append [ | inputlookup perimeter.csv | eval count=0 | fields host count ]
| stats max(_time) AS latest sum(count) AS total BY host
| where total = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

in this way you have to manage the lookup but you have a more affidable control.

Ciao.

Giuseppe