topic Re: How to detect when a host stop sending data to Splunk in Alerting

How to detect when a host stop sending data to Splunk

Lewis — Tue, 17 Aug 2021 02:40:04 GMT

What's the best way to create a base search that will be generic/portable across all clients that will look over a variable period of 45days to identify which hosts have stop sending logs to Splunk? I want to scale it out for about 20,ooo apps. I want to keep the same SPL and use lookups and macros to filter the requirements for each client but, will not be in the base search.

Re: How to detect when a host stop sending data to Splunk

bowesmana — Tue, 17 Aug 2021 02:52:38 GMT

A typical way to determine that something is NOT in Splunk, is to maintain a stored state of the source of truth, e.g. maintain a lookup file of hosts that have-in-the-past sent data to Splunk or hosts that should-always send data to Splunk.

How you maintain that will be somewhat dependent on how source of truth is maintained for each client.

Then if you have your source of truth, the SPL to check for that would be something like

This will then give you a list of hosts that ONLY are found in the csv within the time window you specify. You will of course have to adapt that to fit your requirements.

There is of course the excellent TrackMe app https://splunkbase.splunk.com/app/4621/ which basically does this for you - and lots more.

Re: How to detect when a host stop sending data to Splunk

gjanders — Wed, 18 Aug 2021 06:51:03 GMT

From community slack...

There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warnings

Some helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/