topic Excluding holidays and weekends for Alert and alert if there is 0 events in Splunk Search

Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Mon, 21 Apr 2025 20:08:00 GMT

My search query:

Index=xxx <xxxxxxx>
|eval Date=strftime(_time,"%Y-%m-%d")
| lookup holidays.csv HolidayDate as Date output HolidayDate
| eval should_alert=if((isnull(HolidayDate)), "Yes", "No")
| table Date should_alert
| where should_alert="Yes"

So I've been trying to create an complicated alert.
unfortunately it failed, and is looking for guidance.

The Alert is setup is supposed to fire if there are no results OR more than 1
unless it's the day after a weekend or holiday, in which case, this is

to achieve the alert to look for 0 results OR anything other than 1
I've added below the following:

Trigger conditions:
Number of results
is not equal to 1

so when a date appears on the Muted date(holiday.csv) I want.
turns out it had 0 events that day.
and the 0 events/results triggered the alert and fired on Easter date.

Also when we Mute a dates does it make it return 0 events?
so technically it will still fire on the dates due to my trigger condition, how can we make sure it mutes on the holiday.csv lookup file , and yet alert on 0 events that are not on the holiday.csv

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

livehybrid — Mon, 21 Apr 2025 21:31:05 GMT

Hi @Cheng2Ready

Its hard to write this without seeing the full search but having an alert fire when its !=1 is very limiting, however you might make it work with something like this below.

If there are no results found then you will struggle - so you might need to append an empty |makeresults to ensure that you have atleast 1 event, then you can count the events and check the date:

index=xxx earliest=@d latest=now
| append [|makeresults]
| stats count as event_count
| eval Date=strftime(now(),"%Y-%m-%d") | lookup holidays.csv HolidayDate AS Date OUTPUT HolidayDate | eval wd=strftime(now(),"%w") | eval isWeekend=if(wd=="0" OR wd=="6",1,0)
| where isWeekend=0 AND isnull(HolidayDate) AND event_count!=2

This will return a single event IF its not a weekend/holiday AND the event_count is 2 - Note this is 2 because we're appending a fake result inase there are zero events returned. If zero are returned then it will still append and result in event_count=1 which will then still fire your alert.

You will need to adjust your search to fire when number of results >0 (or !=0)

Does that make sense?

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

gcusello — Tue, 22 Apr 2025 07:14:08 GMT

Hi @Cheng2Ready ,

if you have a lookup containing all the holidays, it's easier to use it as subsearch in the main search, something like this:

index=xxx <xxxxxxx> NOT (date_wday="saturday" OR date_wday="sunday") OR [ | inputlookup holidays.csv | eval date_year=strftime(HolidayDate,"%Y"), date_month=strftime(HolidayDate,"%m"), date_mday=strftime(HolidayDate,"%d") | fields date_year date_month date_mday ]

if you want, in the same way, you could also add a rule for the out of office time (e.g. 18-9).

Ciao.

Giuseppe

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Wed, 23 Apr 2025 17:52:27 GMT

@gcusello in my search query i thought it showed that I have a lookup containing all the holidays that I wanted to have mute.
so yes I do have it.
just wanted to question this line

NOT (dat_wday="saturday" OR date_wday="sunday")

why sat and sunday?
I have my cron schedule to search
0 6 * * 1-5 so its monday-friday

so that should cover it?

could I just

Index=xxx <xxxxxxx>

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Wed, 23 Apr 2025 18:06:21 GMT

@livehybrid
Thank you for the response
yeah I'm still trying to understand its seems like a lot

despite my description of the issue

my run on cron schedule is setup to this

0 6 * * 2-6

tues~saturday.
where Monday and Sunday is excluded to run the search.

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Wed, 23 Apr 2025 18:08:42 GMT

@gcusello Will this fix the issue where it returns "no results"?
my alert would still fire due to this condition

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

gcusello — Thu, 24 Apr 2025 06:42:44 GMT

Hi @Cheng2Ready ,

ok, it's a differenty solution and it's ok.

about your search, you have to decide if you want to use the lookup command (as your original solution) or a subsearch using NOT [...], as my solution, but not the last solution that you shared.

I prefer my solution because it's a best practice to move all the possible search conditions in the main search.

Ciao.

Giuseppe

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Thu, 24 Apr 2025 21:28:35 GMT

Quick question
so lets say we use your query.
When muted on the day of lets say 4/25 and there was an event that happened that day
does the alert the say:
there no results that return therefore it will not fire the alert.

I am trying to figure why my alert fired on of the the dates that my lookup table has chose to mute.
this is my alert settings

cron schedule to search
0 6 * * 1-5 so its monday-friday

but yet the alert fired on a day that it was suppose to mute.
I was wondering could the trigger condition be the root cause?
since there was no results returned and so the trigger alert came to a conclusion no results is also != 1

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

gcusello — Fri, 25 Apr 2025 06:03:33 GMT

Hi @Cheng2Ready ,

you run your alert Monday-Friday, and you filter your results using the above search in this way you will not have results in those days so the alert will not fire.

Ciao.

Giuseppe

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

Cheng2Ready — Wed, 30 Apr 2025 19:58:29 GMT

Thank you @gcusello
appreciate the feedback.
I'm just having trouble understanding why my alert fired when it was not suppose to.
I do not know where to start troubleshooting, but I will accept your answer to the original question

Re: Excluding holidays and weekends for Alert and alert if there is 0 events

gcusello — Thu, 01 May 2025 05:52:59 GMT

hi @Cheng2Ready ,

if you need help, open a new post so more people in Community will be able to help you.

Anyway, start checking what's the condition that fails: if the lookup or the weekday, and then check if it fails every time or some times, and if sometimes, when,

As secondary test, check if it's a border condition: e.g. if the event has timestamp at 23:59:59 or 00:00:00.

Ciao.

Giuseppe