How do I alert when a host stops sending data?

[matt]

What's the best way to create a search to identify which hosts have not sent a syslog message to Splunk in the last 2 days?

[raja8220]

if do search am getting 5 host name in statistics and time if I create alert for it its not triggering ??

AM using age
| metadata index=### type=hosts | eval age = now()-lastTime | where age > 1800 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

[r34220]

I am getting the point where I feel ITSI may not be the right choice for service monitoring for us. How can you rely on ITSI if it can't easily detect if a KPI for a Entity is not getting data in a ITSI Service? What I am finding is if any query returns no results ITSI keeps the last KPI value for an Entity it received. Since Splunk ITSI is used in a large enterprise by several business units, it is imperative that each BU can tell if a Entities not reporing in within a ITSI Service.

In my example, i can tell by the Aggregate Calculation that a host is not reporting in but how can I easily tell which Entity? Only the Entities that have previously stop sending data will show in the Entities section but never drop off once it starts getting data again.

I am using the query

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 300 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Does anyone have a sure way of knowing which Entity has stopped sending data within a ITSI Service?

[raja8220]

if do search am getting 5 host name and time if I create alert for it its not triggering ??

AM using age
| metadata index=### type=hosts | eval age = now()-lastTime | where age > 1800 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

[zward]

I downvoted this post because not relevant to the question, and it is another question -- not an answer or solution to the original question.

[mbain606]

I downvoted this post because this is not suitable for the original question and should not be ranked as an answer because:
1. it is a question about another service
2. the person posted it as an answer when it is really just another question.

[ram_malhotra]

I tried this search an got 0 search results

[ckurtz]

Or that you don't use the main index. Try doing

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 2 | sort age d | convert ctime(lastTime) | fields age,host,lastTime</code

This will search all non-internal indexes and display any hosts that haven't reported in 2 seconds.

[CerielTjuh]

this could mean that you don't have any "lost" hosts

[Lowell · Accepted solution]

Are you talking specifically sourcetype=syslog or just any events from a host? It's easy to do any events from a host with something like this:

| metadata index=main type=hosts | eval age = now()-lastTime | where age > (2*86400) | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Does that work for you?

[bestSplunker]

@matt hey,guys
you can get sourcetype which is stop

|tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype|eval age=now()-lastTime|sort age d|fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S")|eval age=round((age/60/60),1)|search age>=48|eval age=age."hour"