Splunk Search
Highlighted

How do I alert when a host stops sending data?

Splunk Employee
Splunk Employee

What's the best way to create a search to identify which hosts have not sent a syslog message to Splunk in the last 2 days?

Tags (2)
Highlighted

Re: How do I alert when a host stops sending data?

Super Champion

Are you talking specifically sourcetype=syslog or just any events from a host? It's easy to do any events from a host with something like this:

| metadata index=main type=hosts | eval age = now()-lastTime | where age > (2*86400) | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Does that work for you?

View solution in original post

Highlighted

Re: How do I alert when a host stops sending data?

Contributor

@matt hey,guys
you can get sourcetype which is stop

|tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype|eval age=now()-lastTime|sort age d|fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S")|eval age=round((age/60/60),1)|search age>=48|eval age=age."hour"
0 Karma
Highlighted

Re: How do I alert when a host stops sending data?

New Member

I tried this search an got 0 search results

0 Karma
Highlighted

Re: How do I alert when a host stops sending data?

Path Finder

this could mean that you don't have any "lost" hosts

Highlighted

Re: How do I alert when a host stops sending data?

SplunkTrust
SplunkTrust

Or that you don't use the main index. Try doing

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 2 | sort age d | convert ctime(lastTime) | fields age,host,lastTime</code

This will search all non-internal indexes and display any hosts that haven't reported in 2 seconds.

0 Karma
Highlighted

Re: How do I alert when a host stops sending data?

Explorer

I am getting the point where I feel ITSI may not be the right choice for service monitoring for us. How can you rely on ITSI if it can't easily detect if a KPI for a Entity is not getting data in a ITSI Service? What I am finding is if any query returns no results ITSI keeps the last KPI value for an Entity it received. Since Splunk ITSI is used in a large enterprise by several business units, it is imperative that each BU can tell if a Entities not reporing in within a ITSI Service.

In my example, i can tell by the Aggregate Calculation that a host is not reporting in but how can I easily tell which Entity? Only the Entities that have previously stop sending data will show in the Entities section but never drop off once it starts getting data again.

I am using the query

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 300 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

alt text

Does anyone have a sure way of knowing which Entity has stopped sending data within a ITSI Service?

Highlighted

Re: How do I alert when a host stops sending data?

New Member

I downvoted this post because this is not suitable for the original question and should not be ranked as an answer because:
1. it is a question about another service
2. the person posted it as an answer when it is really just another question.

0 Karma
Highlighted

Re: How do I alert when a host stops sending data?

Path Finder

I downvoted this post because not relevant to the question, and it is another question -- not an answer or solution to the original question.

0 Karma
Highlighted

Re: How do I alert when a host stops sending data?

New Member

if do search am getting 5 host name and time if I create alert for it its not triggering ??

AM using age
| metadata index=### type=hosts | eval age = now()-lastTime | where age > 1800 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

0 Karma