Custom Condition Question for Alert

[aferone]

This is the search I am running. I use this search in a dashboard with a single value, color coded:

index=internal source=*metrics.log group=per_index_thruput series!=* | eval totalGB = (kb/1024)/1024 | timechart span=1d sum(totalGB) | rangemap field=sum(totalGB) low=0-13 elevated=13-16 severe=16-9999 default=None

I want an alert to fire if sum(totalGB)> 8. This is what I am trying to use in the "if custom condition is met", but I cannot get the alert to fire:

search sum(totalGB) | where sum(totalGB) > 8

Any help is greatly appreciated. Thanks!

[ryhluc01]

Your server email settings may not be set up yet. All of that has to be in place before you can email reports/notifications and alerts.

https://docs.splunk.com/Documentation/Splunk/latest/Alert/Emailnotification

[Saurabhsood]

thanks eelisio, I know what you are trying to explain. Here I have 1 report in my dashboard as an example. But In real scenario I have more reports in dashboard which I want to schedule based on custom conditions.
Problem is that splunk is not testing for custom conditions and sending emails based on schedule.

[eelisio2]

It sounds like you create more than 1 search and add each to a dashboard as an inline search (not save separately as a Report). Then, you schedule the dashboard to deliver a PDF. Your goal is to have the PDF delivered (or not) based on the result of 1 of the searches in the dashboard.

You edited the ScheduledView_Mydashboard artifact in "Searches, reports, and alerts". Am I correct in understanding that you edit the Search at the top of the form? You replace " | noop" with the first inline search from your dashboard or another search entirely?
And then you edit the Alert Condition to be based on a field from your search.

I have tested this scenario and it works with the Alert Action being "List in Triggered Alerts". I don't have an email server set up. Please try it with List in Triggered Alerts enabled and Send Email not enabled. You can check for the alert under Activity=>Triggered Alerts

[Saurabhsood]

Thanks eelisio for your response.
"List in Triggered Alerts" is working fine. I just checked. But send email is not working. I am expecting that splunk is sending email before checking for custom condition. not sure though.

[sloshburch]

Sanity check: Is Splunk able to send any emails? The sendemail command can be used to validate as well.

Apologies if you already answered this but it does seem peculiar if the alert shows up in the Triggered Alerts but your emails aren't sending. This should clarify if the issue is the search or the hosts mail configuration.

[eelisio2 · Accepted solution]

I would modify the search slightly.

index=_internal source="metrics.log" group=per_index_thruput series!=_ | eval totalGB = (kb/1024)/1024 | timechart span=1d sum(totalGB) as SUMTOTALGB | rangemap field=SUMTOTALGB low=0-13 elevated=13-16 severe=16-9999 default=None

And use the following in the Alert custom criterion:

search SUMTOTALGB > 8

[Saurabhsood]

I have a dashboard with 1 report. I want to schedule this dashboard with custom conditions.
After scheduling the pdf from dashboard, I went to "setting">> "searches,report and alerts" to make custom alert for this dashboard.
I opened _scheduleview_Mydashboard. and in search, i added my below code
"index=main source=winEventlog |stats count(source) as icount"
And then in custom condition for email to trigger, i added "custom condition search" as "search icount>999999"
I am expecting that email will not trigger as icount will be less than my custom condition, but email is getting triggered for dashboard with attached PDF.

Can someone suggest, what's wroung i am doing here in my steps

[eelisio2]

As written, your search will set icount equal to the number of events returned.
The search "index=main source=winEventlog |stats dc(source) as icount" will result in icount being set to 1.

Try skipping the dashboard.
In the search bar add the search, "index=main source=winEventlog |stats dc(source) as icount".
Execute the search.
Click "Save As".
Choose Alert.
You can choose to schedule the alert to execute on a regular interval.
Then,
Edit Trigger Alert When to Custom. Add your condition, "search icount>999999".
Under Trigger Actions, choose Add Action. Select Send Email.
You can choose to include a pdf, csv, link, etc.

[Saurabhsood]

thanks eelisio, I know what you are trying to explain. Here I have 1 report in my dashboard as an example. But In real scenario I have more reports in dashboard which I want to schedule based on custom conditions.
Problem is that splunk is not testing for custom conditions and sending emails based on schedule.

[sloshburch]

Sounds like maybe the search condition in the alert is syntactically wrong? Try adding a 'where' or 'search' filter to the base search of the alert to validate if that at least works.

Essentially what I'm saying is that it sounds like @eelisio2's suggestions all are appropriate and accurate but there might be a subtle detail in your implementation that we're overlooking. Perhaps adding screen shots to your post would help see if there's simply a misunderstanding.

[Saurabhsood]

did not work for me
i have
index="abc"
|dedup ENV
| eval envt=ENV

and condition as
search envt=PROD

Splunk sending emails even if condition does not match

[eelisio2]

Saurabhsood:
I assume that ENV is a field. And the values of the field ENV include "PROD". As written, your search will return the full event that corresponds to the first new value of ENV that it sees. Given the following events:

4/13/2017 PROD The first event
4/13/2017 DEV The second event
4/13/2017 PROD The third event
4/13/2017 ZZZ The fourth event
4/13/2017 DEV The fifth event

Your search would return the first, second and fourth events. The ENV field would have 3 unique values.
i suggest adding a table command to the end of your search to validate that you are getting the results you expect.
index="abc"
|dedup ENV
| eval envt=ENV
| table ENV

[Saurabhsood]

I think i already tried what u suggested but tried one more time but no success. Every time i am getting mail from splunk.

[eelisio2]

Can you test on a small set of events? (Maybe 5-10 events) Can you submit the sample data, the search and the results of your search?

[Saurabhsood]

My search was:
index="abc"
|dedup ENV
| eval envt=ENV
| table envt
Here ENV is a field with only value as "DEV". So ultimately "| table envt" will have only one record as "DEV"
And then in custom condition i tried "search envt=Prod"
i am expecting splunk to stop sending alert for this condition. But it always send me email as an alert.
i think i have tried 100 combinations till now 😞

[tpaulsen]

Super, it works great!

[aferone]

Perfect! Thank you very much!