Alerting

Why is SignalFx not giving Alerts?

tmitra
Observer

I am working in Hxc capture program and currently I am facing an issue in Signalfx. So I am reaching out for help/guidance.

I have created an Alert in signalFx using terraform.

The condition for the alert is that if the signal is less than 1 for a certain time( 2 for warning and 5 for critical),then alert will be triggered.

The code in Signalflow is :

signal = data('HealthCheckStatus', filter=filter('stat', 'upper') and filter('aws_account_id', '823990414917') and filter('aws_tag_Name', 'hxc-staging-health-check')).publish(label='A')

detect(when(signal < 1, '2m')).publish('WARNING[staging]: Route53 health down for 2m')

detect(when(signal < 1, '5m')).publish('CRITICAL[staging]: Route53 health down for 5m')

 

After that we created a condition when the signal came below 1 ( it came to 0).In the Alert graph we can see the same .But instead of actually triggering the alert,it is giving us a preview. (Attached screenshot)

 

The exact Alert message is "Estimated alert count: 1 in 1 hour. Alerts that would have triggered  shown in chart below."

So the bottom line is that Signalfx is not triggering the alert.

The code that I am using for this is : https://github.com/HylandSoftware/tf-cfg-hxc-signalfx-alerts/blob/main/terraform/alerts-route53-heal...

 

Alert Link:https://hyland.signalfx.com/#/detector/v2/FQo6JaWA0AI/edit?detectorSignalFlowEditor=1

 

tmitra_0-1650874746391.png

 

Labels (1)
Tags (2)
0 Karma
Get Updates on the Splunk Community!

Reduce and Transform Your Firewall Data with Splunk Data Management

Managing high-volume firewall data has always been a challenge. Noisy events and verbose traffic logs often ...

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...