Alerting

How to ignore messages logged during application restart?

unitedmarsupial
Path Finder

We have an application, that sends all its log-messages to Splunk (so far so good), and an alert configured to fire, whenever a message with severity above INFO-level is logged.

This works Ok most of the time, except when the application restarts there are multiple such warnings and errors logged by some of its threads. We don't care for these, because the main thread has already announced, that it is shutting down.

How can I phrase the search underlying our alert to exclude any log-entries made after the "I am shutting down" and before the "I started up" ones?

To clarify: we want Splunk to receive all the log-entries, we just don't want the alert to be triggered by those, that are emitted during the program restart...

Labels (1)
0 Karma

unitedmarsupial
Path Finder

@gr0undzer0, no, that's not, what I meant... The downtimes are not scheduled (well, not precisely scheduled), but the application always logs something like "Ok, I'm shutting down", when it is being shut down, and "Started successfully", when it finishes starting back up later.

I'd like my alert to ignore any and all messages logged in between those two. I know, messages can be grouped -- with transaction -- and there are examples for charting how long something took by substracting the start- from the end-timestamp.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that this should be a doable? Just create a SPL query which take care of those unwanted events. Maybe something like this?

  1. Your normal query with events which shows shutdown + start time of that service
  2. sort 0 by _time ?
  3. get shutdown + start time e.g. with eventstats (only one restart exists) or streamstats (more than one restarts within time period)
  4. Drop events which are between start and end time (could be little bit challenging with many restarts 🙂

r. Ismo

0 Karma

Gr0und_Z3r0
Contributor

From the looks of it, you want to suppress alerts during a planned/known outage time window and also at the same time want to have alerts during the normal operational window if the system fails/reboots. Unfortunately Splunk doesn't provide alert suppression windows, your only best bet is to disable alerts during the planned outage window and re-enable them once the activity is completed successfully.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...