Alerting

How to run different alerts/report based on output of some some Alert ?

tarunmalhotra79
Engager

Dear Splunker,

I am trying to implement end to end monitoring where searches have dependecy on multiple lookups and those lookups are dervied from different searches running internally.The idea is to diagnose the situation thouroughly and as early as possible without running all the searches/alerts all the time because there are atleast 60/70 Odd searches in production.
Please look at the below example for more clarity.

search 1
|inputlookup Lookup1
|rename sys_id AS app_sys_id
|lookup Lookup2 parent AS app_sys_id OUTPUTNEW child AS server_sys_id
|mvexpand server_sys_id
|join server_sys_id [inputlookup Lookup3 | rename sys_id AS server_sys_id]
|fields host, server_fqdn, server_status, server_support_group, server_type, server_sox, server_sas, server_admin, server_location, server_environment
|outputlookup Lookup1
Also, as we can see here Lookup1 is dependent on 2 lookups internally (Lookup2,Lookup3)

Search 2
sourcetype=someother_source2|outputlookup Lookup2
here, this sourcetype will eventually create lookup2

Search 3
sourcetype=someother_source3|outputlookup Lookup3
here, this sourcetype will eventually create lookup3

Basically, It's just an example in production I have more than 30 searches in such a manner,I do not want to create 30 alert and running all unnecessary at 30 Odd timings.

I was thinking if i can create something where I will create an alert which will check inside search 1, if the result count is zero or not,if it is zero I should be alerted and then only it should also internally run a report (search 2) to check there if a data is missing or not. If not then move to search 3 to check further, if something is missing then we need to be alerted that data is missing with search 2 or search 3 .It will help me in diagnosing the situation early and also not all the time unnecessary alerts are running on my search head.

Any ideas would be appreciated, I am okay using alerts,report,dashboards,scripts etc.

Again, thanks in advance.

Labels (1)
0 Karma

tarunmalhotra79
Engager

Did anyone get a chance to look into my request, any idea would be highly appreciated?

0 Karma

to4kawa
Ultra Champion
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...