Alerting

How to create a scheduled alert that behaves similar to a real time alert

tyhopping1
Engager

Background: I have one job that runs once a day every day.
Need: I need an alert that triggers when this job runs on the current date.
Issue: I am unable to use "Real time" alerts, so I need a solution that uses Scheduled alerting. I also only want one alert email to be sent off when it is triggered.

Any help is appreciated. Thank you

Tags (2)
0 Karma

maciep
Champion

run it every 5 minutes only, looking back over the previous 5 minutes?

0 Karma

tyhopping1
Engager

That may work. Can you include the fields I should input that will make this happen? Thank you

0 Karma

maciep
Champion

create your search and use the time picker to run it over the past 5 minutes.
save that as an alert
choose to run it on a cron schedule and set that to */5 * * * *

0 Karma

tyhopping1
Engager

This method works. However, the job has has ran for today, but the alert continues to trigger every 5 minutes. Is there a way to trigger it once and stop triggering until the beginning of its to run tomorrow? Thank you

0 Karma

maciep
Champion

I guess that depends on how the search works and what the data looks like. I assumed you'd have a log entry at like 8am that a job ran and then you wouldn't have another until the next day.

So what does your data look like and how are you searching it?

0 Karma

tyhopping1
Engager

That is correct. The job runs once a day. It does not run at the exact same time, but usually a few minutes off.

search:
NameOfJob=ExampleJobName | spath timestamp

| stats earliest(timestamp) as BeginTime , latest(timestamp) as StopTime
count by NameOfJob

data: Uses a timestamp method. Has an early timestamp(beginning) and later timestamp(end). Looking for start and end time of the job. Alerts when job has ended.

0 Karma

maciep
Champion

So what is the timestamp of the event based on then? Why would you see a job that ended say 20 minutes ago if you're only searching for the past 5 minutes?

In any case, you could filter out data where the StopTime is greater than 5 minutes ago. Not sure what format the timestamp is in, but assuming you could use strptime to make it epoch if not already, then something like:

... | where StopTime > relative_time(now(),"-5m")

0 Karma

tyhopping1
Engager

This works. However, I would like to have the email for the alert generate once. Any help is appreciated.

0 Karma

maciep
Champion

can you explain why it is generating more than once? It's hard for me to infer the problem because for any of my data sets, this would work fine. Can you provide an example of when you can alert more than once?

0 Karma

tyhopping1
Engager

It is generating more than once because it is checking every 5 minutes(including after the job has finished).

I need it to alert only once, not multiple times. I need it to alert when the job has ended and that is it.

0 Karma

maciep
Champion

Ok, i don't think we're getting anywhere here. Your search should run every 5 minutes LOOKING BACK 5 over the last minutes ago, e.g. 1:00-1:05, 1:05-1:10, 1:10-1:15, and so on. Why would a job that finished more than 5 minutes ago get returned by your search? I don't understand how that is. You're not explaining that at all.

But if it is returning ALL of your data every 5 minutes for whatever strange reason, did you try filtering by the end time like i showed above? That would exclude any results where the end time is greater than 5 minutes ago. If your job runs once a day, I have no idea how the alert would be firing more than once with that filter in place

Without more details, I can't help any further. You just keep saying the same thing w/o adding anymore info.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...