Hi there, I wanted to get some advice please on a detector creation. For my Cloud Scheduled Functions, I have a custom metric called "job.status" for operational monitoring. This metric has the dimensions: job_id, job_event (START/ERROR/END), and job_event_source (SCHEDULER/JOB). I have Cloud Scheduled Functions that run on cron schedule. For example, 1 service may run its CSFs at: 1pm, 3pm and 5pm daily. I want to create some detectors that: Detector 1: Expected Start Detectors: Detectors assert that a job_event=START metric occurs within a small window around the job's Cron-scheduled start time. For example, the job_event=START occurs around 1pm, 3pm, 5pm daily as per above example. This lets us know if jobs fail to start. Detector 2: Expected Completion Detectors: Detectors assert that a terminal job_event (END/ERROR) metric occurs within the specific window defined by the job's Cron schedule, and/or since the prior job_event=START metric, and the max run time. (E.g., when(hour >= 1 and hour < 2) for a job scheduled only at 1AM and having a max runtime of an hour.) This lets us know if jobs are running too long. Is it possible to incorporate the above detectors using existing logic available in Splunk Observability Cloud? I am finding difficult to achieve the above. It would be great to get some advice. Thanks, Michael
... View more