I am trying to implement end to end monitoring where searches have dependency on multiple lookups and those lookups are derived from different searches running internally.
The idea is to diagnose the situation thoroughly and as early as possible without running all the searches/alerts all the time because there are at least 60/70 Odd searches in production. Please look at the below example for more clarity.
|rename sysid AS appsysid
|lookup Lookup2 parent AS appsysid OUTPUTNEW child AS serversysid
|join serversysid [inputlookup Lookup3 | rename sysid AS serversysid]
|fields host, serverfqdn, serverstatus, serversupportgroup, servertype, serversox, serversas, serveradmin, serverlocation, serverenvironment
Also, as we can see here Lookup1 is dependent on 2 lookups internally (Lookup2, Lookup3):
here, this sourcetype will eventually create lookup2
here, this sourcetype will eventually create lookup3
Basically, It's just an example in production I have more than 30 searches in such a manner, I do not want to create 30 alert and running all unnecessary at 30 Odd timings.
I was thinking if I can create something where I will create an alert which will check inside search 1, if the result count is zero or not, if it is zero I should be alerted and then only it should also internally run a report (search 2) to check there if a data is missing or not. If not then move to search 3 to check further, if something is missing then we need to be alerted that data is missing with search 2 or search 3. It will help me in diagnosing the situation early and also not all the time unnecessary alerts are running on my search head.
Any ideas would be appreciated, I am okay using alerts, reports, dashboards, scripts, etc.