Real Time Alerting Best Practices

[DetectandEngine]

Hello,

I am creating an alert, and want to make sure that the schedule or real time setup sends an email out once the query finds a match. What is the best configuration for an alert to send an email as soon as the criteria of the query matches? 

Thank you! 

[isoutamo]

The best instructions for real time alerts is never ever use those! Usually those generate more issues inside and outside of splunk e.g. in email systems when there are some mistakes in configuration or even any mistakes.

Instead of real time alert you should use scheduled alerts. Just select suitable time schedule for based on individual alert. When you are creating those check if there are regularly some latency when indexing events and if, then adjust earliest and latest based on that.

For sending emails, you could add needed configuration for base splunk email settings or add some alert actions to do it. Personally I prefer to add links to alert into its body, never add real data into it. Time by time there could be some static or similar content. But never send real events outside of splunk.

More instructions can found from community/answers and also alerting manual.

[DetectandEngine]

Thank you for the input to avoid Real Time. When it comes to setting a scheduled alert, what is the best time setting so it alerts as soon as an action that meets the query?  

[ITWhisperer]

Everybody's "best" time is likely to be different depending on their needs, tolerances, reaction time, email infrastructure, network capacity, etc., etc. 

The more frequently you check, the more load you put on your system which will impact other services and users.

You probably should discuss with your stakeholders what would be the maximum time between an event happening and someone being notified about it. Then you have to factor in, how long does it take to get the event into Splunk, how long does it take to get the email out, how long does it take for someone to notice that an email has arrived, how long does it take for them to get into Splunk or reporting system to see the event, etc. Once you have some of these answers, pick a schedule that comes close without overloading your systems and try it out, and be prepared to tweak it. To be honest, it will probably never be right for everyone, but you will probably have to make some compromises.

[PickleRick]

From experience - the initial answer from the requestors will be probably "as soon as possible". Don't fall for that.

[isoutamo]

Usually I ask, How fast you could fix the issue and use that answer to make decision how frequently those alerts should run. Normally I try to use something between 5min to 1h. Of course there could be cases where this must be 1min or longer than 1h.

[PickleRick]

It is also worth remembering that some sources can have their specific characteristics. For example if you're getting WEF-forwarded events with pull-mode, you can receive events in batches every 15 or 30 minutes. For other sources there can be a significant jitter in event delay so they can be backfilling your indexes past the search window. These are just some specific examples to general issues raised earlier. So there is no single answer that fits all possible cases.

[DetectandEngine]

I used the default cron schedule that is listed in Splunk's documentation. What would I need to set so it goes off as soon as there is a match? 

[ITWhisperer]

Assuming you mean you have used a cron schedule for your alert, this means it will execute on that schedule, you just need to configure the alert to trigger when it finds a result worth reporting. This is up to you to decide how to construct your search so that it only triggers when something interesting has happened.