Alerting

outputlookup with no results

MickSheppard
Path Finder

I'm using an outputlookup to generate a list of services for which alerts have been raised in the last 60 minutes. I'm then using this to suppress alerts for the same service within that hour. This works as long as I have a steady stream of different alerts. If I have a lengthy period without any alerts the last alert that I have is suppressed indefinitely.

The problem is that if the search feeding outputlookup doesn't generate any results then the outputlookup doesn't overwrite the file. This means that I never end up with an empty file and consequently I suppress results I shouldn't be.

I'm using this mechanism to suppress the alerts because I want to suppress duplicates for specific events within a given time period and I don't want to define searches for each of the 100+ events I want to trigger a specific alert about. If I could suppress results based on a field value in the result rather than the name of the alert I'd do that.

Tags (1)
0 Karma

ziegfried
Influencer

One option that I can think of is to append a result to the search that generates the information for outputlookup with synthetical values that are never matched by the lookup. Eg.

... | append [ | stats count | fields - count | eval field1="__INVALID__" | eval field2="__INVALID__" | eval ... ] | outputlookup mylookup

Where field1, field2, etc are the columns in your lookup file. This forces the results to be written every time. The downside is the unnecessary entry in the lookup file.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...