Alert safeguards for internal search errors

[danielbb]

I was asked to ask -

Our alerts are relying on various lookups, lookup generators, and other searches. If anything about these underlying layers fail, we have an alert with failing SPL, and these failures are silent, so the alert fails, and we have no idea that it’s because an error in SPL not because there are no events generating them.

Would you ask Splunk Support groups, do we have any option to create an alert action to send us an email whenever a scheduled alert SPL fails due to errors in that SPL? We really need that.

[danielbb]

I created an alert which depends on a lookup. When I deleted the lookup two things took place –

1) The alert invocation was still a success in _internal
2) A message was thrown in _internal - 10-12-2021 10:51:40.961 -0400 ERROR SearchOperator:inputcsv - sid:searchparsetmp_647361294 The lookup table 'lookup_name.csv' requires a .csv or KV store lookup definition.

I don't see a way to correlate the error message with the "successful" invocation of the alert.

[richgalloway]

I recall that at one time Splunk logged an event when a saved search was scheduled and another one when it ran.  The latter is what I expected you to find with the status field.  Perhaps I'm mis-remembering.

[richgalloway]

Create an alert that monitors splunkd.log for instances when your key alerts fail.

index=_internal source=*scheduler.log component=SavedSplunker search_type=scheduled status!=success savedsearch_name IN (<<list of quoted alert names>>)

[yuk]

Hi Rich,

Are you providing this SPL out of your experience, or just guessing?

[richgalloway]

A bit of both.  😉

[danielbb]

Can we search for status as failure?

[richgalloway]

Yes, of course you can and that's what I meant to show in my reply.  See the edited response.