How to setup a Splunk query to alert and send email?

[rtengineer]

Hello.  Try to create a custom alert that does the following.

Monitor Real Time if within certain source "Connection was lost" and then if "Connection has been obtained" is not in the log after 2 minutes, send an email alert out.

I am trying to setup my alert on my search string for Connection  was lost and then put in Trigger alert when "index="main" source="app-api" "Connection has been obtained".

I am receiving an error stating "Cannot parse alert condition. Unknown search command 'index'.." in the Save As Alert window.

Any guidance on how to meet the criteria?

Thanks in advance.

[rtengineer]

I've saved the search as a Report and then selected 'scheduled' and set the parameters as such.

Question is, is there anything with the Schedule, Schedule Window and TimeRange that should be adjusted per the requirement we have?

Thanks

[rtengineer]

Initial Search, index="main" source="ebp-sf-xapi-dev-api" "Connection was lost"

If index="main" source="ebp-sf-xapi-dev-api" "Connection has been obtained" has not been logged 2 minutes after the first query, then send an email alert.

[bowesmana · Accepted solution]

You're going about this the wrong way - using a realtime search to run an alert is bad practice. You should do the checking inside the SPL that is run.

Technically, you need the word "search" before index in the alert if you want to write a custom trigger, but that's not really what it's supposed to be used for

You need to run a search that looks back over 4 minutes for a sequence of events where there is a lost without an obtained message

This may help

index="main" source="ebp-sf-xapi-dev-api" ("Connection was lost" OR "Connection has been obtained")
| rex "Connection (has been|was) (?<state>(obtained|lost))"
| streamstats time_window=2m values(state) as states
| where states="lost" AND mvcount(states)=1

so it searches for both event types - creates a new field called state, which is either lost or obtained, then looks for a case where there is a lost but no obtained within 2 minutes.

Using a 4 minute window means that if you schedule the search to run every 2 minutes and with a search window of 

earliest=-4m@m
latest=@m

if will find cases where there is a lost state at -3:58 and obtained at -1:59.

[rtengineer]

Hello.  Thanks for the input, that does make sense.  As noted I haven't done a lot in terms of Alerts.  So with the search query running this way, do I then save this as a report and create the email alert off of that?  Thanks in advance for your recommendations.

[bowesmana]

Save it as an alert directly, you can set your schedule time range there. as well as the last 4 minute time range.

[rtengineer]

I'm sorry, I'm not following.  I was under the impression I shouldn't set this query as an alert and therefore created the report. 

Just to clarify, the recommendation would be to remove the email action from the report, but instead once pulled up in search, save as an alert.  The question then, am I setting this correctly?  I apologize in advance, but do appreciate your assistance.

[bowesmana]

Sorry for the confusion.

You shouldn't use a custom search string inside the alert to try to achieve what you want to achieve. It's rare you need to do something like that and in your case it wouldn't actually solve your issue correctly.

You should not create real time alerts - unless you like to annoy the Splunk admins 😁

All your SPL to capture the events you want to alert on should be inside your SPL, which you then save as an alert and just trigger on the count of results.

Like reports, alerts can also be scheduled, but also support the alert actions, so if it's an alert you are trying to make, use an alert.

[rtengineer]

Thank you and again apologize for my lack of knowledge here.  So want to re-itterate what you are saying.

I should leave my search as a report because the search criteria needs to be saved somewhere. I'm still unsure how I set this to alert when that search requirement is met.  It looks like you are saying to create an alert from the search, but maybe it's just the parameters I have wrong.  

All your SPL to capture the events you want to alert on should be inside your SPL,  - I read this is my search string, which you then save as an alert and just trigger on the count of results.  So do I not save as a report first?

Thanks in advance and sorry I'm having a hard time understanding your recommendation.  I do appreciate your help.

[bowesmana]

No, you do NOT save as a report, you save as an alert.

An alert is just a different variant of a report. It also contains the search SPL

[bowesmana]

Can you post your exact search query you are using - e.g. screenshot?