Alerting

alerts triggering continuously

CREVITCH
Path Finder

I am saving the following alerts:

"user=* | search failed | dedup _raw"

real time 60 second window. It triggers only when new event with “failed” is detected.

"user=* | search failed | dedup _raw | stats count by user"

real time 60 second window. It alerts continuously

how do I get the second alert to trigger only when a new event is detected?

Tags (2)
0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Hi @CREVITCH,
I think the problem is the transforming command at the end of the second search. This is causing the alert to fire when events are found and when the stats count command returns results.

Our docs state that this can happen with per-result alerts and transforming commands:
"You can create a search to retrieve events from an index. You can also use transforming commands to return results based on processing the retrieved events. A per-result alert triggers in both cases, when the search returns an event or when a transforming command returns results."

See:
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/Defineper-resultalerts

To change the triggering behavior, you can consider adding a trigger condition for the count of events, or perhaps throttling:

"If you have scheduled searches that run frequently and you do not want to be notified for each run, set the throttling controls to suppress the alert to a longer time window.

For real-time searches, if you configure an alert so that it fires once for a trigger condition, you do not need to configure throttling. If the alert fires more than once for the trigger condition, consider throttling to suppress results.

When you configure throttling for a real-time search, start with a throttling period that matches the length of the base search's time window. Expand the throttling period if necessary. This prevents multiple notifications for a given event."

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/ThrottleAlerts

Hope this helps!

0 Karma

CREVITCH
Path Finder

can you show me an example of how to do this?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Quantify Your Splunk Investment Impact: Introducing Savings Metrics to Value Insights

Building on the foundation established in our initial Value Insights releases, we are introducing the Savings ...

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...