Splunk Cloud Platform

Alerts are triggered late

DineshElumalai
Engager

Hello

I have an alert that runs every 2 minutes for the last 40 hours of data. I use five different logs to retrieve the result I need using the join command. The throttle is set on, suppressing the results for 40 hours in order to suppress the repeating alert.

My alert runs perfectly and triggers on time. But every three to four months once, I get the delayed alert for some hours. This issue was repeating for every three to four months, So I had an alternative alert running. Now one of the alert gets delayed for 4 hours and an other one was on time. It makes the alert less reliable.

I started to monitor the triggered alerts in Triggered alerts section. Note: It's a very big query takes 30 seconds.

 May I know the possible reason for this and best practices to avoid this error in future? How to identify the issue? 

Labels (1)
Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

it’s hard to help you without more information about your environment and queries.

You could try to look if this helps https://conf.splunk.com/files/2020/slides/TRU1761C.pdf

There are many more presentations which could help too?

r. Ismo

Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...