Alerting

alert not triggered on expected time and frequency

altink
Builder

The following alert does not trigger in both time and frequency.
The goal was a check every 10 minutes for the last 10 minutes,  but although I performed several tests and the data were there  (alert's search verified) - I could see a single record on form "Triggered Alerts"
only after 16 hours.
Spunk is 7.1

[Access Control Errors]
alert.severity = 4
alert.suppress = 0
alert.track = 1
counttype = number of events
cron_schedule = 0 * * * *
description = Access Control Errors
dispatch.earliest_time = -10m
dispatch.latest_time = now
display.general.type = statistics
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = omega_core_audit
request.ui_dispatch_view = search
search = `mc_unf_md` | search SEO_ACC=1 RETURNCODE!=0


the same alert (and with data from same program) works OK on Splunk 6.4
and with an 1 hour frequency, and with an e-mail action

[Omega CA Access Control Errors]
action.email = 1
action.email.include.trigger_time = 1
action.email.inline = 1
action.email.sendresults = 1
action.email.to = altin.karaulli@unionbank.al
action.email.useNSSubject = 1
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = 0 * * * *
description = Omega CA Access Control Errors
dispatch.earliest_time = -1h
dispatch.latest_time = now
display.events.fields = ["host","source","sourcetype","ACTION_ID","ACTION_NAME","USERNAME","USERHOST","OS_PROCESS","OBJECT_NAME","DB_NAME","POLICY_TYPE_CODE","OS_USER","RETURNCODE","TERMINAL"]
display.events.type = table
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = omega_core_audit
request.ui_dispatch_view = search
search = index=omega_ca SEO_ACC=1 RETURNCODE!=0


Is there anything wrong in the first 7.1 triggered alert case?

best regards
Altin

Labels (1)
Tags (1)
0 Karma
1 Solution

MattibergB
Path Finder

I think you should check you cron schedule, it now only checks at minute 0 every hour for the last -10m so from minute 50 to 00.

You could use */10 * * * *

Check https://crontab.guru/#*/10_*_*_*_*  for more information.

View solution in original post

Tags (1)
0 Karma

MattibergB
Path Finder

I think you should check you cron schedule, it now only checks at minute 0 every hour for the last -10m so from minute 50 to 00.

You could use */10 * * * *

Check https://crontab.guru/#*/10_*_*_*_*  for more information.

Tags (1)
0 Karma

altink
Builder

Thank You @MattibergB 

I followed your crontab advise and set it to run every minute and check for last minute.

...
cron_schedule = */1 * * * *
description = Access Control Errors
dispatch.earliest_time = -1m
dispatch.latest_time = now
...


It did worked correctly. I could see the records in Jobs and Triggered Alerts.

Thank you for the correct answer.

I would still like to know why does it work correctly for the following in 6.4

cron_schedule = 0 * * * *
description = Omega CA Access Control Errors
dispatch.earliest_time = -1h
dispatch.latest_time = now



best regards
Altin

0 Karma

altink
Builder

Thank You

but the schedule is the same on both cases.

cron_schedule = 0 * * * *

but works not only in the first case

Tags (1)
0 Karma

MattibergB
Path Finder

But in the first case you are only looking at data that comes by in minute 50 to 59:59. So if the event that you want to trigger happens at 08:43  your search will not pick it up since it only looks at 08:50 to 08:59.
You changed dispatch.earliest_time to -10m but you did not change the frequency of the search.

In  the second case you are looking at data from the entire previous hour so you would never miss any data.

 

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...