Getting Data In

splunk alert triggering multiple incidents instead of single incident

avi123
Explorer

Hi All,

I have a splunk alert that is having this search query:
index="dcn_b2b_use_case_analytics" sourcetype=lime_process_monitoring
| where BCD_AU_UP_01=0 OR BDC_BA_01=0
| dedup host
| eval failed_processes=mvappend(
if(BCD_AU_UP_01=0, "BCD_AU_UP_01", NULL),
if(BDC_BA_01=0, "BDC_BA_01", NULL)
)
| eval failed_process_list=mvjoin(failed_processes, ", ")
| eval metricLabel="Labware - Services has been stopped in Server--Test Incident--Please Ignore"
| eval metricValue_part1="Hello Application Support team, The below service has been stopped in the server, Service name: "
| eval metricValue_part2=failed_process_list
| eval metricValue_part3=" Server name: "
| eval metricValue_part4=host
| eval metricValue_part5=" Please take the required action to resume the service. Thank you. Regards, Background Service Check Automation Bot"
| eval metricValue=metricValue_part1 + metricValue_part2 + metricValue_part3 + metricValue_part4 + metricValue_part5
| eval querypattern="default"
| eval assignmentgroup="SmartTech Team"
| eval business_service="SmartTech Business Service"
| eval serviceoffering="SmartTech service offering"
| eval Interface="CAB"
| eval urgency=3
| eval impact=3

(Please note: here process status = 0 is failed process and =1 is successful process)

ALERT CONFIG:

Alert type: Scheduled
Cron Expression: */7 * * * *
Expires 24 hours
Trigger Once

Throttle (was checked in checkbox)

Suppress triggering for 30 minutes

When triggered - Alert Action- PTIX SNOWALERT(trigger incident in SNOW)

 

This should trigger only one incident having the Service names and the Server name, but not sure why this alert is triggering three different tickets-please help me correct the alert to trigger single ticket whenever alert is enabled.

Labels (1)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. We don't know what data you're running your search over.

2. Ars you sure you're using dedup right?

3. If you run the search manually, what results does it return?

0 Karma

avi123
Explorer

Hi Will,

I have given this under throttle conditions:

avi123_0-1742831490649.png

 

0 Karma

avi123
Explorer

Hi @livehybrid ,

I had checked the throttle checkbox and enabled Suppress triggering for 30 minutes time to not trigger another incident.

0 Karma

avi123
Explorer

Hi @livehybrid ,

I am getting all the 3 alerts all at the same time. Not sure where the alert is going wrong?

0 Karma

livehybrid
SplunkTrust
SplunkTrust

Hi @avi123 

Do you get the 3 alerts all at the same time, or 7 mins apart?

Regarding the "Suppress results" under the Throttle checkbox, what did you put into this textbox?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...