Splunk Search

How do I alert when a host stops sending data?

matt
Splunk Employee
Splunk Employee

What's the best way to create a search to identify which hosts have not sent a syslog message to Splunk in the last 2 days?

Tags (2)
1 Solution

Lowell
Super Champion

Are you talking specifically sourcetype=syslog or just any events from a host? It's easy to do any events from a host with something like this:

| metadata index=main type=hosts | eval age = now()-lastTime | where age > (2*86400) | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Does that work for you?

View solution in original post

raja8220
New Member

if do search am getting 5 host name in statistics and time if I create alert for it its not triggering ??

AM using age
| metadata index=### type=hosts | eval age = now()-lastTime | where age > 1800 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

0 Karma

r34220
Explorer

I am getting the point where I feel ITSI may not be the right choice for service monitoring for us. How can you rely on ITSI if it can't easily detect if a KPI for a Entity is not getting data in a ITSI Service? What I am finding is if any query returns no results ITSI keeps the last KPI value for an Entity it received. Since Splunk ITSI is used in a large enterprise by several business units, it is imperative that each BU can tell if a Entities not reporing in within a ITSI Service.

In my example, i can tell by the Aggregate Calculation that a host is not reporting in but how can I easily tell which Entity? Only the Entities that have previously stop sending data will show in the Entities section but never drop off once it starts getting data again.

I am using the query

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 300 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

alt text

Does anyone have a sure way of knowing which Entity has stopped sending data within a ITSI Service?

raja8220
New Member

if do search am getting 5 host name and time if I create alert for it its not triggering ??

AM using age
| metadata index=### type=hosts | eval age = now()-lastTime | where age > 1800 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

0 Karma

zward
Path Finder

I downvoted this post because not relevant to the question, and it is another question -- not an answer or solution to the original question.

0 Karma

mbain606
New Member

I downvoted this post because this is not suitable for the original question and should not be ranked as an answer because:
1. it is a question about another service
2. the person posted it as an answer when it is really just another question.

0 Karma

ram_malhotra
New Member

I tried this search an got 0 search results

0 Karma

ckurtz
Path Finder

Or that you don't use the main index. Try doing

| metadata index=* type=hosts | eval age = now()-lastTime | where age > 2 | sort age d | convert ctime(lastTime) | fields age,host,lastTime</code

This will search all non-internal indexes and display any hosts that haven't reported in 2 seconds.

0 Karma

CerielTjuh
Path Finder

this could mean that you don't have any "lost" hosts

Lowell
Super Champion

Are you talking specifically sourcetype=syslog or just any events from a host? It's easy to do any events from a host with something like this:

| metadata index=main type=hosts | eval age = now()-lastTime | where age > (2*86400) | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Does that work for you?

bestSplunker
Contributor

@matt hey,guys
you can get sourcetype which is stop

|tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype|eval age=now()-lastTime|sort age d|fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S")|eval age=round((age/60/60),1)|search age>=48|eval age=age."hour"
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...