Alerting

Alert safeguards for internal search errors

danielbb
Motivator

I was asked to ask -

Our alerts are relying on various lookups, lookup generators, and other searches. If anything about these underlying layers fail, we have an alert with failing SPL, and these failures are silent, so the alert fails, and we have no idea that it’s because an error in SPL not because there are no events generating them.

Would you ask Splunk Support groups, do we have any option to create an alert action to send us an email whenever a scheduled alert SPL fails due to errors in that SPL? We really need that.

 

Labels (1)
Tags (1)
0 Karma

danielbb
Motivator

I created an alert which depends on a lookup. When I deleted the lookup two things took place –

1) The alert invocation was still a success in _internal
2) A message was thrown in _internal - 10-12-2021 10:51:40.961 -0400 ERROR SearchOperator:inputcsv - sid:searchparsetmp_647361294 The lookup table 'lookup_name.csv' requires a .csv or KV store lookup definition.

I don't see a way to correlate the error message with the "successful" invocation of the alert.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I recall that at one time Splunk logged an event when a saved search was scheduled and another one when it ran.  The latter is what I expected you to find with the status field.  Perhaps I'm mis-remembering.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Create an alert that monitors splunkd.log for instances when your key alerts fail.

 

index=_internal source=*scheduler.log component=SavedSplunker search_type=scheduled status!=success savedsearch_name IN (<<list of quoted alert names>>)

 

 

---
If this reply helps you, an upvote would be appreciated.

yuk
Engager

Hi Rich,

Are you providing this SPL out of your experience, or just guessing?

richgalloway
SplunkTrust
SplunkTrust

A bit of both.  😉

---
If this reply helps you, an upvote would be appreciated.

danielbb
Motivator

Can we search for status as failure?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, of course you can and that's what I meant to show in my reply.  See the edited response.

---
If this reply helps you, an upvote would be appreciated.
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...