How do I schedule a recurring search that would alert if an index is missing data feeds from hosts?

[jasonlow]

Hi.

I need to schedule a recurring search that would alert/email me if an index, say "web", is missing data feeds from any of the expected hosts, say web1, web2 and web3, in the past hour. What would this search term be?

In other words, combine three separate scheduled searches into one:

index=web host=web 1 | head 1
index=web host=web 2 | head 1
index=web host=web 3 | head 1

[woodcock]

Did any answer work? Come back and update your question!

[twinspop]

Run this over the last 2 hours (every hour):

| tstats latest(_time) as time where index=web by host
   | eval age=now()-time 
   | where age>3600

if you need standard search features, you can do that too:

index=web some search stuff
   | stats latest(_time) as time by host
   | eval age=now()-time 
   | where age>3600

EDIT: In either case, note that _time is the time presented in the event. You may want to use _indextime instead, which is the time Splunk ingested the event.

[woodcock]

Like this:

|tstats values(host) AS host dc(host) WHERE (index="web" OR index="other") AS hostCount BY index