Getting Data In

splunk alert triggering multiple incidents instead of single incident

avi123
Explorer

Hi All,

I have a splunk alert that is having this search query:
index="dcn_b2b_use_case_analytics" sourcetype=lime_process_monitoring
| where BCD_AU_UP_01=0 OR BDC_BA_01=0
| dedup host
| eval failed_processes=mvappend(
if(BCD_AU_UP_01=0, "BCD_AU_UP_01", NULL),
if(BDC_BA_01=0, "BDC_BA_01", NULL)
)
| eval failed_process_list=mvjoin(failed_processes, ", ")
| eval metricLabel="Labware - Services has been stopped in Server--Test Incident--Please Ignore"
| eval metricValue_part1="Hello Application Support team, The below service has been stopped in the server, Service name: "
| eval metricValue_part2=failed_process_list
| eval metricValue_part3=" Server name: "
| eval metricValue_part4=host
| eval metricValue_part5=" Please take the required action to resume the service. Thank you. Regards, Background Service Check Automation Bot"
| eval metricValue=metricValue_part1 + metricValue_part2 + metricValue_part3 + metricValue_part4 + metricValue_part5
| eval querypattern="default"
| eval assignmentgroup="SmartTech Team"
| eval business_service="SmartTech Business Service"
| eval serviceoffering="SmartTech service offering"
| eval Interface="CAB"
| eval urgency=3
| eval impact=3

(Please note: here process status = 0 is failed process and =1 is successful process)

ALERT CONFIG:

Alert type: Scheduled
Cron Expression: */7 * * * *
Expires 24 hours
Trigger Once

Throttle (was checked in checkbox)

Suppress triggering for 30 minutes

When triggered - Alert Action- PTIX SNOWALERT(trigger incident in SNOW)

 

This should trigger only one incident having the Service names and the Server name, but not sure why this alert is triggering three different tickets-please help me correct the alert to trigger single ticket whenever alert is enabled.

Labels (1)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. We don't know what data you're running your search over.

2. Ars you sure you're using dedup right?

3. If you run the search manually, what results does it return?

0 Karma

avi123
Explorer

Hi Will,

I have given this under throttle conditions:

avi123_0-1742831490649.png

 

0 Karma

avi123
Explorer

Hi @livehybrid ,

I had checked the throttle checkbox and enabled Suppress triggering for 30 minutes time to not trigger another incident.

0 Karma

avi123
Explorer

Hi @livehybrid ,

I am getting all the 3 alerts all at the same time. Not sure where the alert is going wrong?

0 Karma

livehybrid
SplunkTrust
SplunkTrust

Hi @avi123 

Do you get the 3 alerts all at the same time, or 7 mins apart?

Regarding the "Suppress results" under the Throttle checkbox, what did you put into this textbox?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...