Alerting

My alert is not working, how do I troubleshoot?

packet_hunter
Contributor

Looking for a Splunk Jedi Master to shed some light on my failing alert.

I have no problem setting up an alert such as the following (for instance to see when I receive emails from *@splunk.com)

index=mail sourcetype="mail" sender=*@splunk.com| stats values(sender) |sendemail to= ....

alert settings are:
real-time
per-result
send email

The problem is when I create an alert with the same settings for the following:

index=mail 
[search index=mail sourcetype=mail 

[search index=main sourcetype=A eventtype=a suser=* | dedup suser | fields suser| rex field=suser  "<(?<attacker>[\w\d\.\-\@]+)>" | eval sender=lower(attacker) |table sender]

|stats count by internal_message_id | table internal_message_id]

|eval Time=strftime(_time, "%H:%M") | eval Date=strftime(_time, "%F")  | stats list(*)

when I run the second alert manually I get results, and when I add the sendmail to... I get the results mailed, but when I create the alert and verify that it is running 100%, I get nothing.

Does anyone have some suggestions or a check list on how to determine where in the complex subsearch (second code) I went wrong?

Thank you!

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

This is how I would do (these steps are assuming you're creating the alert by saving the search using "Save As" OR updating the existing Alert from Alerts dashboard (Your app-> Navigation Menu-> Alerts)

Search: Use your current search
Alert Type: Scheduled
Time range: -6m@m to -1m@m
Schedule (cron): 1-59/5 * * * *
Trigger Condition: Number of events > 0
Enable Actions: Send Email (setup to, subject,email options per your need)
Action Options : 
When triggered, execute actions : Per result

View solution in original post

0 Karma

somesoni2
Revered Legend

This is how I would do (these steps are assuming you're creating the alert by saving the search using "Save As" OR updating the existing Alert from Alerts dashboard (Your app-> Navigation Menu-> Alerts)

Search: Use your current search
Alert Type: Scheduled
Time range: -6m@m to -1m@m
Schedule (cron): 1-59/5 * * * *
Trigger Condition: Number of events > 0
Enable Actions: Send Email (setup to, subject,email options per your need)
Action Options : 
When triggered, execute actions : Per result
0 Karma

packet_hunter
Contributor

have not tested it yet, waiting for something to roll in and trigger it, but I think you are right (as usual), thank you.

0 Karma

packet_hunter
Contributor

not sure how you saved your alert, (we could be on different versions) but I edited my alert via app: search&reporting>alerts
I think I got everything correct, only difference is the "Time range:" -6m@m to -1m@m

should I change mine, see below?

Settings
Alert [name]
Alert type Scheduled
Run on Cron Schedule
earliest -6m
latest -1m
cron expression 1-59/5****

Trigger Conditions
Trigger alert when Number of Results is greater than 0
Trigger For each result

Thank you

0 Karma

somesoni2
Revered Legend

Any specific reason for running a real-time scheduled search? What I mean to say is that you can run a historical search more frequently instead of a real-time search, provided 1-5 min latency is acceptable to you.

0 Karma

packet_hunter
Contributor

OK, I wanted a real-time search but I obviously must not be doing it right.

Can you send me your suggested settings so I don't muck it up?

Thank you!

0 Karma

somesoni2
Revered Legend

What is the time range/time window you're currently using?

0 Karma

packet_hunter
Contributor

Currently using "all time (real-time)" when I view the alert by Open in Search

I can live with a few minute delay like checking every 5 minutes... just not sure how to set it all up

0 Karma

packet_hunter
Contributor

is this what you are suggesting?

Alert Type:
Real-time. Edit
Trigger Condition:
Number of Results is > 0 in 5 minutes. Edit
Actions:
1 Action
Send email
Edit

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>