Alerting

Unable to get the email alert ? Even when the alert condition is set to trigger when the number of result is less then 0 ?

Hemnaath
Motivator

Hi All, I have used the below query to capture the splunk service status (Up or Down) via splunkd.log. When executed with the time stamp as yesterday we are getting the output. But I want to configure an alert, to run this query for every 15 min and trigger an email alert with the output result.

Query Details :

index=_internal host=hs* sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "ShutdownHandler - shutting down level" OR "TailingProcessor - Shutting down with*" | stats earliest(_time) AS Earliest, values(linecount) as Failures by host | convert ctime(Earliest)|addcoltotals label="Total" labelfield="Total_Number_of_Failures

Below are the configuration steps done to trigger an alert for every 15 min

1) Set Alert type -> Scheduled
2) Time Range --> Run on Cron Scheduled
3) Earliest --> -15m
4) Latest --> now
5) Cron Expression --> */15 * * * *
6) Trigger condition --> Number of Results
7) Trigger if number of results --> if less then 0
8) Email Action --> Send Email
9) Include result --> inline
10) Action option --> Once

Splunk version - 6.0.3

Kindly guide me on how to fix this problem to generate an alert for every 15 mins

thanks in advance.

Tags (1)
0 Karma
1 Solution

inventsekar
Super Champion

the number of alerts can never be less than 0, so, it can never trigger the alert.

Please set the condition as number of alerts if its greater than 0.

6) Trigger condition --> Number of Results
7) Trigger if number of results --> is greater than 0

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.

View solution in original post

0 Karma

inventsekar
Super Champion

the number of alerts can never be less than 0, so, it can never trigger the alert.

Please set the condition as number of alerts if its greater than 0.

6) Trigger condition --> Number of Results
7) Trigger if number of results --> is greater than 0

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

thanks sekar, Initially I had set the trigger condition as number of alerts if it greater than 0 and it did not work. More over this condition will trigger an alert when there is an event generated. So for testing purpose I had set the condition like this "Trigger if number of results --> if less then 0" . So kindly let me know what is going wrong ? why its not triggering the alert.

thanks in advance.

0 Karma

inventsekar
Super Champion

to schedule Job for every two hours in a day -

0 */2 * * *

The cron parameters,
* * * * *, correspond to "minute" "hour" "day-of-month" "month" "day-of-week"

Example expressions
Here are some example cron expressions.

*/5 * * * *       Every 5 minutes.
*/30 * * * *      Every 30 minutes.
0 */12 * * *      Every 12 hours, on the hour.
*/20  * * * 1-5   Every 20 minutes, Monday through Friday.
0 9 1-7 * 1       First Monday of each month, at 9am.
PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

thanks sekar, but I am having few doubts on scheduled saved searches ?
1) where / how to find out whether the scheduled query had fetched the results.
2) I want to schedule a cron job to execute every 2 hours and I had seen your above comments, based on that I had scheduled the cron like this.

++/2 +++   Every 2 hours, on all days of the weeks /months 

Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments.

But it seems it taking it has every 2 min, so kindly correct me on this too if this is not right way to configure the cron job for 2 hours.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi All, Can any one clarify my doubts on Splunk scheduled reports.

1) where/how to find out whether the scheduled query had fetched the results after executing the query.

2) I had configured to schedule a cron job to execute every 2 hours and based on the above comments, like this

++/2 +++ Every 2 hours, on all days of the weeks /months.
Kindly correct me if this not the right way to schedule the cron job.
Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments.
thanks in advance.

0 Karma

inventsekar
Super Champion

Hi Hemnaath,
1. on your splunk, settings--> Searches, reports, and alerts, then find your saved search.
on your saved search, under "actions" row, you can see "View recent | Run | Advanced edit | Clone | Move | Delete". click "View recent".
on this "Searches, reports, and alerts" page, you can see a column "Alerts", which says the alerts count. when you click View Recent, you can see how many events was fetched from your scheduled search.

  1. 0 */2 * * * Every 2 hours, at the 0th min. * */2 * * * - i am not sure whether we can have a "*" for min.
PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

thanks sekar.

0 Karma

inventsekar
Super Champion
  1. when you run this query, do you get results?!?!
  2. From other splunk alerts, do you receive email alerts? I mean, pls make sure the email notifications are working fine
  3. Do you have splunk admin access or user access
PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

yes, I am able to get the result while executing the query and even I was getting the email alerts, when Cron Expression alone was set to this value /15 * * * * but not getting the latest output.
so when I include **Time Range --> Earliest= -15m Latest=now
*, i am not getting the email alert.

my requirement is to monitor the splunk services status for every 15 mins and should alert in case of failure of the service ( by capturing the splunkd.log) by executing the above query. But i am not whether the problem is with the splunk query or with the splunk configuration.

Kindly guide me to fix this issue. thanks in advance.

0 Karma

ddrillic
Ultra Champion

If that's the case, run the query from the command line and choose the past 15 minutes. Do you get results?

0 Karma

Hemnaath
Motivator

thanks all for throwing some lights on this issue. I am getting an email alert when I had set the following time range as we had some splunk service failure on last weekend.

Time Range --> Earliest= -3d Latest=now.

But when I had set the time range to -15m and latest=now, I am not getting any alert, so waiting for some failure to happen, so that I can validate the query.

0 Karma

inventsekar
Super Champion

Hi Hemnaath,
for cron schedule, did you set it like this -
*/15 * * * *
if you have set it as "/15 *", please update it as above.

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

Hi Sekar, thanks for your inputs on this. It worked, It triggered email notification alerts to our mail id, when an Splunk service went down.

Yes I had already set the cron Job like what you had mentioned above.

**/15 * * * **

0 Karma

inventsekar
Super Champion

Great..nice to know that it worked. Can you please mark it as an accepted answer.

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

hey can you tell me where/how to check whether the saved search job is running or not in splunk. As I had set an saved search report to execute the query for every 2 hours but I had not set a email notification for the same. In this case how/where to check.

thanks in advance.

0 Karma

inventsekar
Super Champion

on your splunk, settings--> Searches, reports, and alerts, then find your saved search.
on your saved search, under "actions" row, you can see "View recent | Run | Advanced edit | Clone | Move | Delete". click "View recent".

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

Hemnaath
Motivator

thanks sekar, I need to set the cron job to execute the script every two hours and I have set the cron job but not sure why its not triggering the alert. Kindly guide we whether cron job is set correctly or not .

Schedule Job for every two hours in a day.
00 /2 ***

thanks in advance.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...