My alert is not working, how do I troubleshoot?

[packet_hunter]

Looking for a Splunk Jedi Master to shed some light on my failing alert.

I have no problem setting up an alert such as the following (for instance to see when I receive emails from *@splunk.com)

index=mail sourcetype="mail" sender=*@splunk.com| stats values(sender) |sendemail to= ....

alert settings are:
real-time
per-result
send email

The problem is when I create an alert with the same settings for the following:

index=mail 
[search index=mail sourcetype=mail 

[search index=main sourcetype=A eventtype=a suser=* | dedup suser | fields suser| rex field=suser  "<(?<attacker>[\w\d\.\-\@]+)>" | eval sender=lower(attacker) |table sender]

|stats count by internal_message_id | table internal_message_id]

|eval Time=strftime(_time, "%H:%M") | eval Date=strftime(_time, "%F")  | stats list(*)

when I run the second alert manually I get results, and when I add the sendmail to... I get the results mailed, but when I create the alert and verify that it is running 100%, I get nothing.

Does anyone have some suggestions or a check list on how to determine where in the complex subsearch (second code) I went wrong?

Thank you!

[somesoni2 · Accepted solution]

This is how I would do (these steps are assuming you're creating the alert by saving the search using "Save As" OR updating the existing Alert from Alerts dashboard (Your app-> Navigation Menu-> Alerts)

Search: Use your current search
Alert Type: Scheduled
Time range: -6m@m to -1m@m
Schedule (cron): 1-59/5 * * * *
Trigger Condition: Number of events > 0
Enable Actions: Send Email (setup to, subject,email options per your need)
Action Options : 
When triggered, execute actions : Per result

[packet_hunter]

have not tested it yet, waiting for something to roll in and trigger it, but I think you are right (as usual), thank you.

[packet_hunter]

not sure how you saved your alert, (we could be on different versions) but I edited my alert via app: search&reporting>alerts
I think I got everything correct, only difference is the "Time range:" -6m@m to -1m@m

should I change mine, see below?

Settings
Alert [name]
Alert type Scheduled
Run on Cron Schedule
earliest -6m
latest -1m
cron expression 1-59/5****

Trigger Conditions
Trigger alert when Number of Results is greater than 0
Trigger For each result

Thank you

[somesoni2]

Any specific reason for running a real-time scheduled search? What I mean to say is that you can run a historical search more frequently instead of a real-time search, provided 1-5 min latency is acceptable to you.

[packet_hunter]

OK, I wanted a real-time search but I obviously must not be doing it right.

Can you send me your suggested settings so I don't muck it up?

Thank you!

[somesoni2]

What is the time range/time window you're currently using?

[packet_hunter]

Currently using "all time (real-time)" when I view the alert by Open in Search

I can live with a few minute delay like checking every 5 minutes... just not sure how to set it all up

[packet_hunter]

is this what you are suggesting?

Alert Type:
Real-time. Edit
Trigger Condition:
Number of Results is > 0 in 5 minutes. Edit
Actions:
1 Action
Send email
Edit