Unable to get the email alert ? Even when the alert condition is set to trigger when the number of result is less then 0 ?

[Hemnaath]

Hi All, I have used the below query to capture the splunk service status (Up or Down) via splunkd.log. When executed with the time stamp as yesterday we are getting the output. But I want to configure an alert, to run this query for every 15 min and trigger an email alert with the output result.

Query Details :

index=_internal host=hs* sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "ShutdownHandler - shutting down level" OR "TailingProcessor - Shutting down with*" | stats earliest(_time) AS Earliest, values(linecount) as Failures by host | convert ctime(Earliest)|addcoltotals label="Total" labelfield="Total_Number_of_Failures

Below are the configuration steps done to trigger an alert for every 15 min

1) Set Alert type -> Scheduled
2) Time Range --> Run on Cron Scheduled
3) Earliest --> -15m
4) Latest --> now
5) Cron Expression --> */15 * * * *
6) Trigger condition --> Number of Results
7) Trigger if number of results --> if less then 0
8) Email Action --> Send Email
9) Include result --> inline
10) Action option --> Once

Splunk version - 6.0.3

Kindly guide me on how to fix this problem to generate an alert for every 15 mins

thanks in advance.

[inventsekar · Accepted solution]

the number of alerts can never be less than 0, so, it can never trigger the alert.

Please set the condition as number of alerts if its greater than 0.

6) Trigger condition --> Number of Results
7) Trigger if number of results --> is greater than 0

[Hemnaath]

thanks sekar, Initially I had set the trigger condition as number of alerts if it greater than 0 and it did not work. More over this condition will trigger an alert when there is an event generated. So for testing purpose I had set the condition like this "Trigger if number of results --> if less then 0" . So kindly let me know what is going wrong ? why its not triggering the alert.

thanks in advance.

[inventsekar]

to schedule Job for every two hours in a day -

0 */2 * * *

The cron parameters,
* * * * *, correspond to "minute" "hour" "day-of-month" "month" "day-of-week"

Example expressions
Here are some example cron expressions.

*/5 * * * *       Every 5 minutes.
*/30 * * * *      Every 30 minutes.
0 */12 * * *      Every 12 hours, on the hour.
*/20  * * * 1-5   Every 20 minutes, Monday through Friday.
0 9 1-7 * 1       First Monday of each month, at 9am.

[Hemnaath]

thanks sekar, but I am having few doubts on scheduled saved searches ?
1) where / how to find out whether the scheduled query had fetched the results.
2) I want to schedule a cron job to execute every 2 hours and I had seen your above comments, based on that I had scheduled the cron like this.

++/2 +++   Every 2 hours, on all days of the weeks /months 

Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments.

But it seems it taking it has every 2 min, so kindly correct me on this too if this is not right way to configure the cron job for 2 hours.

thanks in advance.

[Hemnaath]

Hi All, Can any one clarify my doubts on Splunk scheduled reports.

1) where/how to find out whether the scheduled query had fetched the results after executing the query.

2) I had configured to schedule a cron job to execute every 2 hours and based on the above comments, like this

++/2 +++ Every 2 hours, on all days of the weeks /months.
Kindly correct me if this not the right way to schedule the cron job.
Note - Instead of star symbol I have used + to describe cron set up as I am unable to use star symbol in comments.
thanks in advance.

[inventsekar]

Hi Hemnaath,
1. on your splunk, settings--> Searches, reports, and alerts, then find your saved search.
on your saved search, under "actions" row, you can see "View recent | Run | Advanced edit | Clone | Move | Delete". click "View recent".
on this "Searches, reports, and alerts" page, you can see a column "Alerts", which says the alerts count. when you click View Recent, you can see how many events was fetched from your scheduled search.

1. 0 */2 * * * Every 2 hours, at the 0th min. * */2 * * * - i am not sure whether we can have a "*" for min.

[Hemnaath]

thanks sekar.

[inventsekar]

1. when you run this query, do you get results?!?!
2. From other splunk alerts, do you receive email alerts? I mean, pls make sure the email notifications are working fine
3. Do you have splunk admin access or user access

[Hemnaath]

yes, I am able to get the result while executing the query and even I was getting the email alerts, when Cron Expression alone was set to this value /15 * * * * but not getting the latest output.
so when I include **Time Range --> Earliest= -15m Latest=now*, i am not getting the email alert.

my requirement is to monitor the splunk services status for every 15 mins and should alert in case of failure of the service ( by capturing the splunkd.log) by executing the above query. But i am not whether the problem is with the splunk query or with the splunk configuration.

Kindly guide me to fix this issue. thanks in advance.

[ddrillic]

If that's the case, run the query from the command line and choose the past 15 minutes. Do you get results?

[Hemnaath]

thanks all for throwing some lights on this issue. I am getting an email alert when I had set the following time range as we had some splunk service failure on last weekend.

Time Range --> Earliest= -3d Latest=now.

But when I had set the time range to -15m and latest=now, I am not getting any alert, so waiting for some failure to happen, so that I can validate the query.

[inventsekar]

Hi Hemnaath,
for cron schedule, did you set it like this -
*/15 * * * *
if you have set it as "/15 *", please update it as above.

[Hemnaath]

Hi Sekar, thanks for your inputs on this. It worked, It triggered email notification alerts to our mail id, when an Splunk service went down.

Yes I had already set the cron Job like what you had mentioned above.

**/15 * * * **

[inventsekar]

Great..nice to know that it worked. Can you please mark it as an accepted answer.

[Hemnaath]

hey can you tell me where/how to check whether the saved search job is running or not in splunk. As I had set an saved search report to execute the query for every 2 hours but I had not set a email notification for the same. In this case how/where to check.

thanks in advance.

[inventsekar]

on your splunk, settings--> Searches, reports, and alerts, then find your saved search.
on your saved search, under "actions" row, you can see "View recent | Run | Advanced edit | Clone | Move | Delete". click "View recent".

[Hemnaath]

thanks sekar, I need to set the cron job to execute the script every two hours and I have set the cron job but not sure why its not triggering the alert. Kindly guide we whether cron job is set correctly or not .

Schedule Job for every two hours in a day.
00 /2 ***

thanks in advance.