Alerting

How to schedule an alert to run at 11:00PM PST (23:00) every day and trigger an email if the number of events is less than 15?

Explorer

Hi All,

Here is my requirement:

I want to create a alert if the number of events is less than 15. It should trigger email everyday at 23.00 (11.00pm pst)

Below is the search:

Index=abc sourcetype=pqr "ERROR"

The cron job I used is 00 23 star star star ( 00 23 * * *)

starttime: -24h
finishtime: now

Here is the actual problem:
If I schedule it for every 5 minutes or 10 minutes, the email is triggering and working fine (testing purpose). Once if I change for 23.00hrs(11.00 pm pst), the alert is not working and not triggering an email and there are no other searches running at that time.

Please help.

Below is my savedsearches.conf:

[ERROR]
action.email = 1
action.email.inline = 1
action.email.sendresults = 1
action.email.to = abc@gmail.com
alert.digest_mode = True
alert.suppress = 0
alert.track = 1
auto_summarize.dispatch.earliest_time = -1d@h
counttype = number of events
cron_schedule = 00 23***
dispatch.earliest_time = -24h
dispatch.latest_time = now
enableSched = 1
quantity = 15
relation = less than
schedule_window = 5
search = index=abc sourcetype=pqr "ERROR"

alt text

0 Karma
1 Solution

Revered Legend

Well, see the search head utilization and you can either increase resources OR add more SH accordingly. For this issue, I guess all these searches are running with default splunk-system-user accound (means owner=nobody), and it's hitting it's limit big time. You should change the owner to specific user accounts needing this data, preferably power user/admin type of account so that their job quota is used to run these searches instead of default splunk user.

View solution in original post

0 Karma

Revered Legend

Well, see the search head utilization and you can either increase resources OR add more SH accordingly. For this issue, I guess all these searches are running with default splunk-system-user accound (means owner=nobody), and it's hitting it's limit big time. You should change the owner to specific user accounts needing this data, preferably power user/admin type of account so that their job quota is used to run these searches instead of default splunk user.

View solution in original post

0 Karma

Explorer

Thanks somesoni,

It is working now after changing the owner to admin.Thanks

0 Karma

Revered Legend

If you run this search manually, how many events do you get?? Also, when you run every 5 or 10 minutes, do you keep the same start time and finish time??

Index=abc sourcetype=pqr "ERROR" earliest=-24h
0 Karma

Explorer

Hi somesoni,
Yes, If i schedule a cron for every 5 min or 10 min , i am using the earliest time(-24h) and finishtime(now).
and it should trigger email irrespective of the number of events. because my condition is less than.If there is no events also it should trigger email.

0 Karma

Revered Legend

Agree.. Can you try to set the value for "Schedule Window" to 0 (as per screenshot it's 5)?

0 Karma

Revered Legend

Also, can you check if the your saved search was executed and didn't trigger alert?

index=_internal sourcetype=scheduler savedsearch_name="Error"
0 Karma

Explorer

Hi somesoni,

Yes i made window 0 and i checked .but same result.If i check splunkd it is showing below error

"02-18-2016 12:10:52.760 -0800 WARN SHPMaster - Search not executed: Your maximum number of concurrent searches has been reached. usage=2828 quota=150 user=splunk-system-user. for search: nobody"

For this i increased srchjobsquota also no change.should we increase any crores?
Thanks

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!