Alerting

How to disable splunk alert from 1 Am to 6.30 Am .for particular time?

karthi2809
Builder

How to customize alert should not trigger for particular time from 1 AM to 6.30AM?

Tags (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi karthi2809
insert in your main search this condition:

date_hour>=1 AND (date_hour<7 AND date_minute<31)

if this condition must run only in some days of the week insert an additionan condition with date_wday.
Bye.
Giuseppe

karthi2809
Builder

Hi Cusello,

index=docker fatal NOT date_hour>=1 AND (date_hour<7 AND date_minute<31)

will this query works.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi karthi2809,
the NOT condition must be all in parenthesis:

index=docker fatal NOT (date_hour>=1 AND (date_hour<7 AND date_minute<31))

otherwise only the first condition is called.

Bye.
Giuseppe

0 Karma

sureshkumaar
Path Finder

Hi @gcusello - Looks like this condition isn't working in the query where we use "stats" part of query for CPU, Memory, Disk space alerts. Kindly suggest

 

NOT (date_wday=monday AND date_hour>=00 AND (date_hour<07))

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

the object of date_hour is a number, "00" and "07" aren't number, did you tried:

NOT (date_wday=monday AND date_hour>=0 AND (date_hour<7))

?

Ciao.

Giuseppe

sureshkumaar
Path Finder

Hi @gcusello  - I tried now but still results are getting shows in the events view

NOT (date_wday=monday AND date_hour>=0 AND (date_hour<7))

What I could observe here is when the stats command in place _time field isn't working in Splunk so the condition we use here looks for the query where _time filed is effectively used.

Kindly let me know your thoughts/observations on this

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

sorry I was wrong, after a stats command, you haven't more the date_wday and date_hour fields but you have only the fields in the stats.

Could you share your search?

 Ciao.

Giuseppe

0 Karma

sureshkumaar
Path Finder

Hi @gcusello - i cannot share the actual/complete search but something with Splunk fields i can share

 

source=*cpu* cpu_load_percent>75 NOT (date_wday=monday AND date_hour>=0 AND (date_hour<7)) | stats latest(cpu_load_percent) as cpu_max_usage by _time, host | table _time host cpu_max_usage

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

try something like this:

source=*cpu* cpu_load_percent>75 NOT (date_wday="monday" date_hour<7) 
| stats latest(cpu_load_percent) as cpu_max_usage by _time, host 
| table _time host cpu_max_usage

In this way, you exclude alerts fired on monday between 0 and 6.

Ciao.

Giuseppe

sureshkumaar
Path Finder

Hi @gcusello - Still not working and results are getting displayed

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

what's the time period of your search?

in this way you exclude events from 00 to 7.00 but alert is running so if you have events at 23.55 and you run your alert at 0.30 the alert is fired!

Do you want to exclude events from 00 to 7.00 or avoid to run alert?

If you don't want to execute the alert between 00 and 7.00 you have to configure your cron.

Ciao.

Giuseppe

0 Karma

sureshkumaar
Path Finder

Hi @gcusello  - The time period of my search is "Last 30 minutes", But for testing purpose to check the working of that "NOT" condition I modified to search "Today" as time range to look for results for that time returning or not.

But the results are showing the stats value between the time 1 AM - 3 AMto disable splunk alert.png

 

NOT (date_wday=monday AND date_hour>=0 AND (date_hour<7))

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

it's a really strange behaviour!

Anyway, only for debugging, what does happen if you:

  • don't use NOT (first try),
  • don't use date_wday="monday" (second try)

?

Ciao.

Giuseppe

0 Karma

sureshkumaar
Path Finder

Hi @gcusello - I tried those steps but still not working. I have attached the snapshot for reference.

Initially with actual query

Later without NOT and then also without date_wday=tuesdayWithout NOT condition.png

source=*cpu* cpu_load_percent>75 NOT (date_wday=tuesday AND date_hour>=0 AND (date_hour<3)) | stats latest(cpu_load_percent) as cpu_max_usage by _time, host | table _time host cpu_max_usage

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

it seems that you haven't the fields date_hour and date_wday.

Could you run your main search without the date_hour and date_wday conditions and without the following stats and table commands?

in other words

source=*cpu* cpu_load_percent>75

I want to see if date_hour and date_wday are in the interesting fields .

If they aren't (and it's strange) you have to build them using eval.

Ciao.

Giuseppe

0 Karma

sureshkumaar
Path Finder

Hi @gcusello  - Found that issue lies with docker related servers where default date/time fields are missing from the docker side itself so we have reached out to outcold solutions

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sureshkumaar,

good for you, tell us if you need more help.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are ppreciated by all the Contributors 😉

0 Karma

karthi2809
Builder

Hi cusello,

The query is not working the alert is triggering at that time.

Thanks,
Karthigeyan

0 Karma

niketn
Legend

@karthi2809, can you share your alert query?

Can you display date_hour as well in your alert? Is it possible that there is difference in the timezone for your data and your local time zone?

Try the following:

index=docker fatal (date_hour<1 OR date_hour>6) OR (date_hour=6 AND date_minute>30)
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

karthi2809
Builder

Thanks cusello

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...