Alerting

How to ignore messages logged during application restart?

unitedmarsupial
Path Finder

We have an application, that sends all its log-messages to Splunk (so far so good), and an alert configured to fire, whenever a message with severity above INFO-level is logged.

This works Ok most of the time, except when the application restarts there are multiple such warnings and errors logged by some of its threads. We don't care for these, because the main thread has already announced, that it is shutting down.

How can I phrase the search underlying our alert to exclude any log-entries made after the "I am shutting down" and before the "I started up" ones?

To clarify: we want Splunk to receive all the log-entries, we just don't want the alert to be triggered by those, that are emitted during the program restart...

Labels (1)
0 Karma

unitedmarsupial
Path Finder

@gr0undzer0, no, that's not, what I meant... The downtimes are not scheduled (well, not precisely scheduled), but the application always logs something like "Ok, I'm shutting down", when it is being shut down, and "Started successfully", when it finishes starting back up later.

I'd like my alert to ignore any and all messages logged in between those two. I know, messages can be grouped -- with transaction -- and there are examples for charting how long something took by substracting the start- from the end-timestamp.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that this should be a doable? Just create a SPL query which take care of those unwanted events. Maybe something like this?

  1. Your normal query with events which shows shutdown + start time of that service
  2. sort 0 by _time ?
  3. get shutdown + start time e.g. with eventstats (only one restart exists) or streamstats (more than one restarts within time period)
  4. Drop events which are between start and end time (could be little bit challenging with many restarts 🙂

r. Ismo

0 Karma

Gr0und_Z3r0
Communicator

From the looks of it, you want to suppress alerts during a planned/known outage time window and also at the same time want to have alerts during the normal operational window if the system fails/reboots. Unfortunately Splunk doesn't provide alert suppression windows, your only best bet is to disable alerts during the planned outage window and re-enable them once the activity is completed successfully.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...