Knowledge Management

Two time period search with summary index or kvstore

koshyk
Super Champion

We have a rare query from a team and situation is
- The team needs to immediately get an alert (within 5 minutes)
- The team don't want to miss an alert (even if there is a network delay or indexing delay or some issues)
- Also the alert cannot be duplicated if it is already alerted
- Alerting is done on the event-time (and not indextime)

So an event comes in real-time, then there is no problem.
But if there are network issues, and assume the event got delayed by 6 minutes they need a sweep up of such delayed alert, so it will be still alerted, but NOT shown duplicate

I thought of an option to
- have two searches (SavedSearch1 => one which runs every 5mins, searching for previous 5 mins) & (SavedSearch2=> which runs every 5 mins but sweeps events in last 60mins).
- Summary index the SavedSearch1 and SavedSearch2 should compare if it is already there in the summary index using same time

So my query is
Have you guys done this similar situation? Is there any other better option?

0 Karma

adonio
Ultra Champion

hmmm
and why the following wont work?
_index_earliest=-4m@m _index_latest=-1m@m .... all your query ... | stats <whatever> by _time _indextime ... | where or eval if needed ...
then run the alert every 3 minutes and add your relevant condition in the alert dialog boxes.
you are not supposed to miss a thing
oh, dont forget to mark the search / alert as highest priority

hope it helps

koshyk
Super Champion

that's a fair point. Let me test this out practically and see if anything get's missed (or any scenario's will get missed)

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...