Alerting

5.0.3.1 -- Scheduled search creating lots of directories in dispatch despite setting expiration to 1s

the_wolverine
Champion

Conditions:
1) Scheduled search that runs every minute and writes to summary index.

2) Additionally, configured to alert "always" and email to someone@email.address.
3) Expiration set to custom time, 1 second.
4) savedsearches.conf config for this search clearly shows 1s expiration value:

alert.expires = 1s

Expectation -- only 1 (or a few) directories related to this search in dispatch. Its ok if the cleanup job can't run often enough to keep it exactly at 1 second, even a few minutes would be fine.

Observation -- over 1000 directories generated by this scheduled search.

0 Karma

the_wolverine
Champion

Suspicion - This scheduled search is affected because it is considered an alert and different ttl applies to the generated "alert" artifacts.

Of more specific suspicion, in etc/system/default/alert_actions.conf:

ttl         = 86400

(86400 seconds = 24 hours)

Or, there's a bug.

sowings
Splunk Employee
Splunk Employee

A job which triggers an alert action takes on the TTL of that action.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...

Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend ...

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward ...