Alerting

My alert is not working, how do I troubleshoot?

packet_hunter
Contributor

Looking for a Splunk Jedi Master to shed some light on my failing alert.

I have no problem setting up an alert such as the following (for instance to see when I receive emails from *@splunk.com)

index=mail sourcetype="mail" sender=*@splunk.com| stats values(sender) |sendemail to= ....

alert settings are:
real-time
per-result
send email

The problem is when I create an alert with the same settings for the following:

index=mail 
[search index=mail sourcetype=mail 

[search index=main sourcetype=A eventtype=a suser=* | dedup suser | fields suser| rex field=suser  "<(?<attacker>[\w\d\.\-\@]+)>" | eval sender=lower(attacker) |table sender]

|stats count by internal_message_id | table internal_message_id]

|eval Time=strftime(_time, "%H:%M") | eval Date=strftime(_time, "%F")  | stats list(*)

when I run the second alert manually I get results, and when I add the sendmail to... I get the results mailed, but when I create the alert and verify that it is running 100%, I get nothing.

Does anyone have some suggestions or a check list on how to determine where in the complex subsearch (second code) I went wrong?

Thank you!

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

This is how I would do (these steps are assuming you're creating the alert by saving the search using "Save As" OR updating the existing Alert from Alerts dashboard (Your app-> Navigation Menu-> Alerts)

Search: Use your current search
Alert Type: Scheduled
Time range: -6m@m to -1m@m
Schedule (cron): 1-59/5 * * * *
Trigger Condition: Number of events > 0
Enable Actions: Send Email (setup to, subject,email options per your need)
Action Options : 
When triggered, execute actions : Per result

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

This is how I would do (these steps are assuming you're creating the alert by saving the search using "Save As" OR updating the existing Alert from Alerts dashboard (Your app-> Navigation Menu-> Alerts)

Search: Use your current search
Alert Type: Scheduled
Time range: -6m@m to -1m@m
Schedule (cron): 1-59/5 * * * *
Trigger Condition: Number of events > 0
Enable Actions: Send Email (setup to, subject,email options per your need)
Action Options : 
When triggered, execute actions : Per result
0 Karma

packet_hunter
Contributor

have not tested it yet, waiting for something to roll in and trigger it, but I think you are right (as usual), thank you.

0 Karma

packet_hunter
Contributor

not sure how you saved your alert, (we could be on different versions) but I edited my alert via app: search&reporting>alerts
I think I got everything correct, only difference is the "Time range:" -6m@m to -1m@m

should I change mine, see below?

Settings
Alert [name]
Alert type Scheduled
Run on Cron Schedule
earliest -6m
latest -1m
cron expression 1-59/5****

Trigger Conditions
Trigger alert when Number of Results is greater than 0
Trigger For each result

Thank you

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Any specific reason for running a real-time scheduled search? What I mean to say is that you can run a historical search more frequently instead of a real-time search, provided 1-5 min latency is acceptable to you.

0 Karma

packet_hunter
Contributor

OK, I wanted a real-time search but I obviously must not be doing it right.

Can you send me your suggested settings so I don't muck it up?

Thank you!

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is the time range/time window you're currently using?

0 Karma

packet_hunter
Contributor

Currently using "all time (real-time)" when I view the alert by Open in Search

I can live with a few minute delay like checking every 5 minutes... just not sure how to set it all up

0 Karma

packet_hunter
Contributor

is this what you are suggesting?

Alert Type:
Real-time. Edit
Trigger Condition:
Number of Results is > 0 in 5 minutes. Edit
Actions:
1 Action
Send email
Edit

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...