Other Usage

Why is Alert action not triggering when using real time?

theprophet01
Explorer

-I am running an alert which is not triggering email actions when using real-time option.   The alert is used to  search for hosts which have not sent logs in the last 5 minutes.

-For example, I shut down a host for testing and wait 5 minutes. I then manually use the search string and specify time frame (e.g. last 15 minutes)- I am able to obtain results.

However,  even though the same search was configured in the form of an alert running in real time, it produces no results nor does it trigger an email.

Here is the search I am using:

 

 

index=* | stats max(_time) as latest by host | eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") | fields - latest | where recent = 0 | rename host AS Host, realLatest AS "Latest Timestamp" | table Host, "Latest Timestamp"

 

 

 

Labels (3)
Tags (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @theprophet01,

using a search like yours with Real-Time it isn't a good idea because you are using one CPU only for this search reducing the resources of your global Splunk infrastructure.

It's better to schedure a search e.g. every 5 minutes, so, when running is finished, the search releases the CPU for other jobs.

In addition, your search could be optimized to reduce the execution time and the CPU use:

| tstats max(_time) AS latest count BY host
| eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") 
| where recent = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

At least, using this search you find only the hosts that didn't send logs in the last 5 minutes, but that sent logs in the previous 10 minutes (using a timeframe of 15 minutes); if your host doesn't send logs for 15 minutes you loose this information.

The best approach is having a lookup containing all the hosts to monitor (called e.g. perimeter.csv) containing at least one column (host) and running a search like the following:

| tstats max(_time) AS latest count BY host
| append [ | inputlookup perimeter.csv | eval count=0 | fields host count ]
| stats max(_time) AS latest sum(count) AS total BY host
| where total = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

in this way you have to manage the lookup but you have a more affidable control.

Ciao.

Giuseppe

View solution in original post

theprophet01
Explorer

That worked perfect! I also used the 5 min scheduled search as suggested using a cron schedule. Thank you @gcusello you sir are indeed a legend!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @theprophet01,

using a search like yours with Real-Time it isn't a good idea because you are using one CPU only for this search reducing the resources of your global Splunk infrastructure.

It's better to schedure a search e.g. every 5 minutes, so, when running is finished, the search releases the CPU for other jobs.

In addition, your search could be optimized to reduce the execution time and the CPU use:

| tstats max(_time) AS latest count BY host
| eval recent= if(latest > relative_time(now(),"-5m"),1,0). realLatest = strftime(latest, "%Y-%M-%D %H%M%S") 
| where recent = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

At least, using this search you find only the hosts that didn't send logs in the last 5 minutes, but that sent logs in the previous 10 minutes (using a timeframe of 15 minutes); if your host doesn't send logs for 15 minutes you loose this information.

The best approach is having a lookup containing all the hosts to monitor (called e.g. perimeter.csv) containing at least one column (host) and running a search like the following:

| tstats max(_time) AS latest count BY host
| append [ | inputlookup perimeter.csv | eval count=0 | fields host count ]
| stats max(_time) AS latest sum(count) AS total BY host
| where total = 0 
| rename host AS Host, realLatest AS "Latest Timestamp" 
| table Host, "Latest Timestamp"

in this way you have to manage the lookup but you have a more affidable control.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...