Alerting

Alert triggers even tho it should not

nicxso
Engager

Hi,

I created an alert, which counts the total events per sourcetype and compares it with the total events from last week. The alert should trigger when the total events of today has doubled or more than last week. I don't know why it's wrong. Also is there a way to optimize this search. It takes around 1 to 2 mins to execute this search

index=caas_oracle_virtual_directory_* sourcetype="oracle_virtual_directory:*" earliest=-7d@d latest=-7d 
| stats count as event_last_week by index, sourcetype 
| join type=left sourcetype
    [ 
    | search index=caas_oracle_virtual_directory_* sourcetype="oracle_virtual_directory:*" earliest=@d latest=now 
    | stats count as event_today by index, sourcetype 
    | fields sourcetype, event_today] 
| eval time_now = strftime(now(), "%d.%m.%y %H:%M:%S") 
| eval double = (event_last_week*2) 
| where double > event_today 
| table time_now, index, sourcetype, event_last_week, event_today, double 
| rename time_now as "Time", index as "Index" sourcetype as "Source Type", event_last_week as "Total Events Last Week", event_today as "Event Today", double as "Threshold"

And here the output of the alarm:
alt text

Thanks for your help!

0 Karma

adonio
Ultra Champion

hello there,

hope i understand the question
try this out:

| tstats count as event_count where earliest=-7d@d latest=now index=caas_oracle_virtual_directory_* sourcetype="oracle_virtual_directory:*" by index sourcetype _time span=1d
| bucket _time span=1d 
| stats last(_time) AS last_time sum(event_count) AS per_day_count by _time, sourcetype 
| eval last_week = if(last_time > exact(relative_time(now(),"-8d@d")) AND last_time <= exact(relative_time(now(),"-7d@d")) , per_day_count ,"0") 
| eval today = if(last_time > exact(relative_time(now(),"-1d@d")) AND last_time <= exact(relative_time(now(),"-0d@d")) , per_day_count ,"0") 
| stats sum(today) AS today sum(last_week) AS last_week by sourcetype
| eval double_last_week = last_week * 2
| eval alert = if(today > double_last_week,1,0)

hope it helps

0 Karma

nicxso
Engager

@adnonio

I'll give this a try tomorrow. I also created a similar alert to the first one but that alert only triggers when the total events from today has halved or more compared to last week. So if I would change this part to this.

| eval half_last_week = last_week / 2 
 | eval alert = if(today < half_last_week,1,0)

Would this work and will the output be the same as the one I had ?

0 Karma

adonio
Ultra Champion

supposed to

0 Karma

nicxso
Engager

Cause for example in need to count all the data from today eg. 03/22/2019 12:00:00 AM until now (11:06:00)
and compare last week ago wich would be 03/15/2019 12:00:00 AM until the now (11:06:00). Thats how i would like to compare it and the output should also include the index

Thanks

0 Karma

adonio
Ultra Champion

you can modify the earliest and latest to match your needs =

0 Karma

nicxso
Engager

Just asking, does the search compare the data from the current time now compared to the current time last week ago ?
Or else it works

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...