Alerting

How to search for events based on another event generated time

apujari
Explorer

I have an event that is monitoring a host and triggers an alert when the host is down. I have used below search and its working fine:

index=main url=hosturl title=hostname response_code!=200

Now, I need to generate another alert when the host comes up next. For this, I am tracking the alert (for down host) time when it got triggered and then searching for the host monitoring event which got triggered after that to check if the host is up/down. Below is the search query which I tried and need help on.

index="_internal" sourcetype="scheduler" thread_id="AlertNotifier*" alert_actions="email" savedsearch_name="Host-Tracking" | stats latest(_time) as alerttime | append [search index=main title=hostname response_code=200 earliest=alerttime]

Labels (1)
0 Karma
1 Solution

apujari
Explorer

Thanks everyone for the suggestions. This is how I configured the search query:

index=main url=hosturl title=hostname response_code=200 earliest=[search index="_internal" sourcetype="scheduler" thread_id="AlertNotifier\" alert_actions="email" savedsearch_name="Host-Tracking" | stats latest(_time) as alerttime | append [| makeresults | eval alerttime = now()] | return $alerttime]*

which is working fine as per my requirements.

View solution in original post

0 Karma

apujari
Explorer

Thanks everyone for the suggestions. This is how I configured the search query:

index=main url=hosturl title=hostname response_code=200 earliest=[search index="_internal" sourcetype="scheduler" thread_id="AlertNotifier\" alert_actions="email" savedsearch_name="Host-Tracking" | stats latest(_time) as alerttime | append [| makeresults | eval alerttime = now()] | return $alerttime]*

which is working fine as per my requirements.

0 Karma

to4kawa
Ultra Champion

good job.
please accept yours.

p.s. have you read my blog?

0 Karma

apujari
Explorer

Yes, it was really helpful for me and gave enough idea to configure the above query. Thank you so much for the help.

to4kawa
Ultra Champion

@apujari
great! It was hard to read.

Happy splunking!

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Here's one way. Since we're not certain from your code what the host name might be in either kind of record, or how often you might want to run this, we wrote this is pseudocode for you.

(index="_internal" sourcetype="scheduler" thread_id="AlertNotifier*" alert_actions="email" savedsearch_name="Host-Tracking" 
earliest = however long you want to go back for alerts)

OR  (index=main title=hostname response_code=200
earliest = however often you want to run this )  

| eval myHost=case(this/is/a/scheduler/event, host field from scheduler record, 
                   this/is/a/main/index/event, host field from main record)
| eval alert_time=case(this/is/a/scheduler/event, _time)
| eval hostup_time=case(this/is/a/main/index/event, _time)
| stats max(alert_time) as alert_time max(hostup_time) as hostup_time by myHost 
| where hostup_time > alert_time
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

What are the fields that need to be matched between the records of each kind?

Does "title" the actual name of the host?

0 Karma

apujari
Explorer

Any suggestions please..

0 Karma

to4kawa
Ultra Champion

https://qiita.com/toshikawa/items/38e57c6f2b0514db109b
sorry, all text is japanese.
please translate them.
but SPL is useful for you.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...