Alerting

How to create an alert if no data is generated from a host?

ashari
Explorer

I want to run a search in splunk to find out that all the devices attached to the splunk server are generating logs. If I dont hear from a device I should receive a alert.

thanks

Tags (1)

martin_mueller
SplunkTrust
SplunkTrust

Here's a quick way to check for hosts not having sent data in a while:

| tstats latest(_time) as latest where index=* by host | where latest < relative_time(now(), "-1d")

Change the -1d according to your needs, run the above search over at least twice the scheduled interval.

koshyk
Super Champion

We had similar siutation and a heartbeat from the other system comes once in every 5 minute. So I'm doing that as a "saved Search" and run every 1 minute and checks data for the previous 6 minutes to see if there is a heartbeat. Else alert.

The logic is:

index=myindex earliest=-6m@s <your_some_more_Search> | stats count as COUNT_HEARTBEAT | where COUNT_HEARTBEAT=0 
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi ashari,

check out the Splunk Deployment Monitor there you can find some nice saved searches related to this topic ... keyword: missing forwarders

hope that helps ...

cheers, MuS

0 Karma

MuS
SplunkTrust
SplunkTrust

sure, although there are a lot of search macros in the saved searches. If you look at the searches you will get an idea how it can be done.
Otherwise use a lookup file which contains all your forwarder hostnames and check if all of them were seen in the past x minutes. Take a look at this http://answers.splunk.com/answers/73268/search-for-hosts-in-a-lookup-but-not-in-splunk

ashari
Explorer

can we do this without splunk deployment monitor app.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Build the Future of Agentic AI: Join the Splunk Agentic Ops Hackathon

AI is changing how teams investigate incidents, detect threats, automate workflows, and build intelligent ...

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...