Alerting

Need Help w/ Multiple Alert Emails

joeybroesky
Path Finder

Need assistance figuring out why we are receiving multiple email alerts. We are trying to setup email alerts for Office365 Service Messages.

Search String

index="o365data" sourcetype="o365:service:message"  Id=*
 | where Classification == "Incident" AND Severity == "Sev2"
 | spath Messages{} output=Messages
 | spath WorkloadDisplayName
 | spath Id
 | spath Status
 | stats values(WorkloadDisplayName) as WorkloadDisplayName values(Id) as Id values(Status) as Status by Messages
 | spath input=Messages
 | eval PublishedTime=strptime(PublishedTime, "%Y-%m-%dT%H:%M:%S.%NZ")
 | eval CorrectPublished=PublishedTime+25200
 | where MessageText != "A post-incident report has been published."
 | stats count by CorrectPublished Id WorkloadDisplayName MessageText Status 
 | sort - CorrectPublished
 | dedup CorrectPublished
 | fields - count
 | eval CorrectPublished=strftime(CorrectPublished,"%Y/%m/%d %T")
 | fields - PublishedTime
 | dedup Id
 | table CorrectPublished Id WorkloadDisplayName MessageText Status
 | rename CorrectPublished as "Published", Id as "ID", WorkloadDisplayName as "Workload", MessageText as "Details"

Alert Settings
Alert Type: Real-time
Expires: 48 Hour(s)
Trigger alert when: Per-Result
Throtte: yes
Suppress results containing field value: *
Suppress triggering for: 2 minute(s)
When triggered: Send email

Emails Received
alt text

alt text

Labels (2)
Tags (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Try changing Trigger alert when: Per-Result to Trigger alert when: Once.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try changing Trigger alert when: Per-Result to Trigger alert when: Once.

---
If this reply helps you, Karma would be appreciated.
0 Karma

joeybroesky
Path Finder

Thanks Rich! I'll test this and report back!

0 Karma

joeybroesky
Path Finder

@richgalloway my apologies for the delay, I should be able to verify on the next alert.

0 Karma

joeybroesky
Path Finder

This did the trick. Thanks again Rich!!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@joeybroesky If your problem is solved then please accept the answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...