Hi,
We have a custom search that should alert when a critical host, that we have defined in the search, is missing. The issue we're having is that we haven't been alerted on some of the hosts not having logs because the last time they had any logs was an age ago. When I change earliest=-1d to earliest=-1y the hosts I want appear but the search takes a longer time.
Is there a way to make it so for every host value specified, a stats line is created where I can fillnull the fields as appropriate?
Here is the search:
| tstats prestats=true count where index="*", (host="host01" OR host="host02" OR host="host_01" OR host="host_02") earliest=-1d latest=now by index, host, _time span=1h
| eval period=if(_time>relative_time(now(),"-1d"),"current","last")
| chart count over host by period
| eval missing=if(last>0 AND (isnull(current) OR current=0), 1, 0)
| where missing=1
| sort - current, missing
| rename current as still_logging, last as old_logs, missing as is_missing
Hi
I think that you will found answer from these links:
There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...Some helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/
r. Ismo