Search to Identify when a host stops sending logs to Splunk

[Makinde]

Hello,

I have this search string to identify hosts that have stopped sending logs to Splunk, however the search string below identifies every hosts that has ever stopped sending logs, however I want only hosts that have not sent any logs in the past 3 days. What do I need to change in this search string to get that number?

|metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

[twinspop · Accepted solution]

I would probably use tstats instead of metadata, and just set the time selector to last 3 days:

| tstats latest(_time) as lastTime by host | 
eval age=now()-lastTime | 
search age > 86400 | 
sort age d | 
convert ctime(lastTime) | 
fields age,host,lastTime

Metadata can yield unexpected results when you set a timeframe.

EDIT: changed to days from hours. Doh!

[Makinde]

Hi Twinspop,

Thanks for this new search, it appears to work better than the Metadata. Just curious, If I understand properly, this search looks at logs as far back as I specify in my time selector and identifies hosts that haven't reported in the time specified in the search age criteria (search age > 86400) in this case more than a day?

Is that what this search does?

[twinspop]

Correct. The tstats command will follow your time restraint. This command will initially find all hosts that have logged any data in the last 3 days in any index. The filtering will then only show those that stopped more 86400 seconds ago.

[muebel]

Hi Makinde, something like this should work (could probably drop the 86400 down closer to now to be more inclusive)

|metadata type=hosts | eval age = now() - lastTime | search age > 86400 AND age < 259200 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Please let me know if this answers your question!

[Makinde]

Hi Muebel,

The new search definitely makes a change in my results however I noticed it doesn't identify hosts that stopped sending logs older than 3 days ago. So say a host stopped sending logs last month and it hasn't sent any logs up until now that won't show up in this search result.