Splunk Search

What is a good query to monitor for someone sending too many alerts?

MikeBertelsen
Communicator

I received an email from ES techs that someone had sent over 128k alerts to the same address in a 24 hour period.
I tracked it down to two private alerts and disabled them.
Researching further those emailed alerts were just among those successfully sent. Because a lot of people did not get their alerts or scheduled reports for that day.

Here is a sampling of the error message from internals:
04-09-2019 09:15:07.768 -0500 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://TheSearchHeadServer:8000/app/ALL_my_sales/@go?sid=scheduler
karlA28BCL_ZGlnaXRhbF9zYWxlcwRMD559a15d8ba081a9e5_at_1554819300_52888" "ssname=Null Pointer" "graceful=True" "trigger_time=1554819306" results_file="/opt/splunk/var/run/splunk/dispatch/schedulerkarlA28BCL_ZGlnaXRhbF9zYWxlcw_RMDL559a15d8ba081a9e5_at_1554819300_52888/results.csv.gz"': ERROR:root:(452, '4.3.1 Insufficient system storage', u'SplunkSH@gmail.com') while sending mail to: karlA28BCL@gmail.com

0 Karma

MikeBertelsen
Communicator

Here is the query I have so far:
host=SplunkSH index=_internal "-0500 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/search/bin/sendemail.py" "Insufficient system storage'" "while sending mail to:"

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...