topic Re: Alert safeguards for internal search errors in Alerting

Alert safeguards for internal search errors

danielbb — Fri, 08 Oct 2021 19:55:53 GMT

I was asked to ask -

Our alerts are relying on various lookups, lookup generators, and other searches. If anything about these underlying layers fail, we have an alert with failing SPL, and these failures are silent, so the alert fails, and we have no idea that it’s because an error in SPL not because there are no events generating them.

Would you ask Splunk Support groups, do we have any option to create an alert action to send us an email whenever a scheduled alert SPL fails due to errors in that SPL? We really need that.

Re: Alert safeguards for internal search errors

richgalloway — Sat, 09 Oct 2021 00:05:02 GMT

Create an alert that monitors splunkd.log for instances when your key alerts fail.

index=_internal source=*scheduler.log component=SavedSplunker search_type=scheduled status!=success savedsearch_name IN (<<list of quoted alert names>>)

Re: Alert safeguards for internal search errors

danielbb — Fri, 08 Oct 2021 20:52:06 GMT

Can we search for status as failure?

Re: Alert safeguards for internal search errors

richgalloway — Sat, 09 Oct 2021 00:05:46 GMT

Yes, of course you can and that's what I meant to show in my reply. See the edited response.

Re: Alert safeguards for internal search errors

yuk — Mon, 11 Oct 2021 09:25:16 GMT

Hi Rich,

Are you providing this SPL out of your experience, or just guessing?

Re: Alert safeguards for internal search errors

richgalloway — Mon, 11 Oct 2021 12:31:35 GMT

A bit of both. 😉

Re: Alert safeguards for internal search errors

danielbb — Tue, 12 Oct 2021 15:57:54 GMT

I created an alert which depends on a lookup. When I deleted the lookup two things took place –

1) The alert invocation was still a success in _internal
2) A message was thrown in _internal - 10-12-2021 10:51:40.961 -0400 ERROR SearchOperator:inputcsv - sid:searchparsetmp_647361294 The lookup table 'lookup_name.csv' requires a .csv or KV store lookup definition.

I don't see a way to correlate the error message with the "successful" invocation of the alert.

Re: Alert safeguards for internal search errors

richgalloway — Tue, 12 Oct 2021 20:20:55 GMT

I recall that at one time Splunk logged an event when a saved search was scheduled and another one when it ran. The latter is what I expected you to find with the status field. Perhaps I'm mis-remembering.