How to ignore messages logged during application restart?

[unitedmarsupial]

We have an application, that sends all its log-messages to Splunk (so far so good), and an alert configured to fire, whenever a message with severity above INFO-level is logged.

This works Ok most of the time, except when the application restarts there are multiple such warnings and errors logged by some of its threads. We don't care for these, because the main thread has already announced, that it is shutting down.

How can I phrase the search underlying our alert to exclude any log-entries made after the "I am shutting down" and before the "I started up" ones?

To clarify: we want Splunk to receive all the log-entries, we just don't want the alert to be triggered by those, that are emitted during the program restart...

[unitedmarsupial]

@Anonymous, no, that's not, what I meant... The downtimes are not scheduled (well, not precisely scheduled), but the application always logs something like "Ok, I'm shutting down", when it is being shut down, and "Started successfully", when it finishes starting back up later.

I'd like my alert to ignore any and all messages logged in between those two. I know, messages can be grouped -- with transaction -- and there are examples for charting how long something took by substracting the start- from the end-timestamp.

[isoutamo]

Hi

I think that this should be a doable? Just create a SPL query which take care of those unwanted events. Maybe something like this?

1. Your normal query with events which shows shutdown + start time of that service
2. sort 0 by _time ?
3. get shutdown + start time e.g. with eventstats (only one restart exists) or streamstats (more than one restarts within time period)
4. Drop events which are between start and end time (could be little bit challenging with many restarts 🙂

r. Ismo

[Gr0und_Z3r0]

From the looks of it, you want to suppress alerts during a planned/known outage time window and also at the same time want to have alerts during the normal operational window if the system fails/reboots. Unfortunately Splunk doesn't provide alert suppression windows, your only best bet is to disable alerts during the planned outage window and re-enable them once the activity is completed successfully.