Splunk Search

Generating alert based on condition

Ashutosh_30
Loves-to-Learn

Hi All ,

i have configured alerts for the search below:

index="ebs_red_0" host="dev-obiee-ux0*" source="/obiee_12c/app/oracle/product/12212/user_projects/domains/bi/nodemanager/nodemanager.log" "waiting for the process to die"

Output :

8/3/21
9:38:11.000 AM
dev-obiee-ux08The server 'obips2' with process id 12714242 is no longer alive; waiting for the process to die.obips2obiee:nodemanager:logAug 3, 2021 5:38:11 AM EDT

 

but sometimes when my server process dies it restarts automatically within a 60 seconds which can be described as :

index="ebs_red_0" host="dev-obiee-ux0*" source="/obiee_12c/app/oracle/product/12212/user_projects/domains/bi/nodemanager/nodemanager.log" "is running now"

Output : 

8/3/21
9:39:27.000 AM
dev-obiee-ux08The server 'obis2' is running now.obis2obiee:nodemanager:logAug 3, 2021 5:39:27 AM EDT

 

So i want to write the search query in a way so that i generate alert only if the server process dies and doesn't come up again within 120 seconds.


the five fields used in the search are :

_time, host ,Message ,OBIEE_Comp, sourcetype ,time

 

and to generate the alert the OBIEE_Comp needs to be same 


Labels (1)
0 Karma

Ashutosh_30
Loves-to-Learn

This works but the problem is i'm getting all the results in single row.
for this :
index="ebs_red_0" host="dev-obiee-ux0*" source="/obiee_12c/app/oracle/product/12212/user_projects/domains/bi/nodemanager/nodemanager.log" "Server was killed" OR "waiting for the process to die"
| stats earliest(_time) as _time,values(time) as time,values(host) as host, values(Message) as Message, values(OBIEE_Comp) as OBIEE_Comp , values(sourcetype) as sourcetype, range(_time) as time_gap | where time_gap > 120 | table time, host ,Message ,OBIEE_Comp, sourcetype ,time_gap

Ashutosh_30_0-1628668603893.png


i want to get every record in a separate row




0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @Ashutosh_30 

Try this and schedule it for every 5 mins,

index="ebs_red_0" host="dev-obiee-ux0*" source="/obiee_12c/app/oracle/product/12212/user_projects/domains/bi/nodemanager/nodemanager.log" "waiting for the process to die" OR "is running now"
| stats earliest(_time) as time, values(Message) as message, values(OBIEE_Comp) as OBIEE_Comp , values(sourcetype) as sourcetype, range(_time) as time_gap
| where time_gap > 120
| table _time, host ,Message ,OBIEE_Comp, sourcetype ,time_gap

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...