Splunk Cloud Platform

Adjustment of Suppression Time for P1 Splunk Alerts

raghunandan1
Engager

Hi Team,

We have P1 Splunk alerts generated based on event ID: 12320 triggered from the following servers:

scwdxxxxx0009
scwdxxxxx0008
scwpxxxxx0002
scwpxxxxx0001
Recently, we identified that we have a 24-hour suppression time for the alert, which led to a critical incident. To address this issue, the user has requested a reduction in the suppression time for the alert.

The goal is to eliminate suppression unless the previous triggered alert is still open. If there are no open P1 tickets for event ID: 12320, there should not be any suppression of the generation of new tickets.

Current Alert Configuration
We have one alert in Splunk, and we are using the following query:

Splunk query:

index=winevent sourcetype="WinEvent:*" ((host="scwpxxxxx0001*" OR host="scwdxxxxx0008*" OR host="scwdxxxxx0009*" OR host="scwpxxxxx0002*") AND (EventCode=12320))
| eval assignment_group = "ABC IT - Computing Services"
| eval host=lower(mvindex(split(host,"."),0))
| eval correlation_id=strftime(_time,"%Y-%m-%d %H:%M:%S").":".host
| eval short_description=case((host="scwpxxxxx0001" OR host="scwdxxxxx0008"),"Microsoft AAD Proxy Connector - Prod not able to connect due to network issues.",(host="scwdxxxxx0009" OR host="scwpxxxxx0002"),"Microsoft AAD Proxy Connector - Dev not able to connect due to network issues.", 1=1, 0 )
| eval category="Application", subcategory="Repair/Fix", contact_type="Event", state=4, ci=host, customer="no573", impact=1, urgency=1, description="Event Code ".EventCode." encountered on host ".host." at ".strftime(_time,"%m/%d/%Y %H:%M:%S %Z")." SourceName:".SourceName." Log Name: ".LogName." TaskCategory:".TaskCategory." Message=".Message." Ticket generated on SNOW at ".strftime(now(),"%m/%d/%Y %H:%M:%S %Z")
| table host, short_description, assignment_group, impact, urgency, category, subcategory, description, ci, correlation_id
Alert Type
Scheduled
Schedule
Run on cron schedule: */30 * * * * (every 30 minutes)
Time Range
Last 4 hours
Expiration
24 hours
Throttle Enabled
Suppress results containing field value: host, EventCode
Suppress triggering for: 24 hours
Trigger Actions
ServiceNow Incident Integration


How we can suppress the alert as per the requirement. Please help us here.

Thank you.

Labels (2)
Tags (2)
0 Karma

raghunandan1
Engager

Please help on that

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...