We have some scheduled jobs that I recently noticed on the Jobs page have error messages ("max_mem_usage_mb has been reached" in our case). I wasn't aware that these searches were not producing the correct results due to running out of memory. Is there a way to set up an email alert to be notified when scheduled jobs have error messages? I'm able to find the messages in var/run/splunk/dispatch, but that data doesn't appear to be searchable (like in _internal for instance) in which case I could set up a scheduled search to detect these occurrences. In the absence of the error messages being searchable, how can we be notified?
Also, I am able to find the job run in index=_internal (sourcetype = scheduler) , but the entry says "status=success" even though the Job page lists an error.
... View more