topic splunk alert triggering multiple incidents instead of single incident in Getting Data In

splunk alert triggering multiple incidents instead of single incident

avi123 — Mon, 24 Mar 2025 14:54:29 GMT

Hi All,

I have a splunk alert that is having this search query:
index="dcn_b2b_use_case_analytics" sourcetype=lime_process_monitoring
| where BCD_AU_UP_01=0 OR BDC_BA_01=0
| dedup host
| eval failed_processes=mvappend(
if(BCD_AU_UP_01=0, "BCD_AU_UP_01", NULL),
if(BDC_BA_01=0, "BDC_BA_01", NULL)
)
| eval failed_process_list=mvjoin(failed_processes, ", ")
| eval metricLabel="Labware - Services has been stopped in Server--Test Incident--Please Ignore"
| eval metricValue_part1="Hello Application Support team, The below service has been stopped in the server, Service name: "
| eval metricValue_part2=failed_process_list
| eval metricValue_part3=" Server name: "
| eval metricValue_part4=host
| eval metricValue_part5=" Please take the required action to resume the service. Thank you. Regards, Background Service Check Automation Bot"
| eval metricValue=metricValue_part1 + metricValue_part2 + metricValue_part3 + metricValue_part4 + metricValue_part5
| eval querypattern="default"
| eval assignmentgroup="SmartTech Team"
| eval business_service="SmartTech Business Service"
| eval serviceoffering="SmartTech service offering"
| eval Interface="CAB"
| eval urgency=3
| eval impact=3

(Please note: here process status = 0 is failed process and =1 is successful process)

ALERT CONFIG:

Alert type: Scheduled
Cron Expression: */7 * * * *
Expires 24 hours
Trigger Once

Throttle (was checked in checkbox)

Suppress triggering for 30 minutes

When triggered - Alert Action- PTIX SNOWALERT(trigger incident in SNOW)

This should trigger only one incident having the Service names and the Server name, but not sure why this alert is triggering three different tickets-please help me correct the alert to trigger single ticket whenever alert is enabled.

Re: splunk alert triggering multiple incidents instead of single incident

livehybrid — Mon, 24 Mar 2025 15:17:18 GMT

Hi @avi123

Do you get the 3 alerts all at the same time, or 7 mins apart?

Regarding the "Suppress results" under the Throttle checkbox, what did you put into this textbox?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: splunk alert triggering multiple incidents instead of single incident

avi123 — Mon, 24 Mar 2025 15:24:02 GMT

Hi @livehybrid ,

I am getting all the 3 alerts all at the same time. Not sure where the alert is going wrong?

Re: splunk alert triggering multiple incidents instead of single incident

avi123 — Mon, 24 Mar 2025 15:31:06 GMT

Hi @livehybrid ,

I had checked the throttle checkbox and enabled Suppress triggering for 30 minutes time to not trigger another incident.

Re: splunk alert triggering multiple incidents instead of single incident

avi123 — Mon, 24 Mar 2025 15:52:12 GMT

Hi Will,

I have given this under throttle conditions:

Re: splunk alert triggering multiple incidents instead of single incident

PickleRick — Mon, 24 Mar 2025 22:07:21 GMT

1. We don't know what data you're running your search over.

2. Ars you sure you're using dedup right?

3. If you run the search manually, what results does it return?