I have a usecase where I need to compare daily reports and set up an alert on the deltas.
Usecase is: Hosts that are reporting yesterday, but not today.
I have the search as this.
index=os source=daily reporting hosts reporting=yes | table host, ip | dedup host.
I'm using set command , NOT and passing time parameters but I'm not getting the right result.
set diff [search index=os source=daily reporting hosts reporting=yes earliest=-3d latest=-2d | table host, ip | dedup host] [search index=os source=daily reporting hosts reporting=yes earliest=-2d latest=-1d | table host, ip | dedup host] | table host .
[search index=os source=daily reporting hosts reporting=yes earliest=-3d latest=-2d | table host, ip | dedup host] NOT [search index=os source=daily reporting hosts reporting=yes earliest=-2d latest=-1d | table host, ip | dedup host] | table host .
Help me if you have any suggestions to better deal with this usecase or any change to the query.
you could run a search similar to this:
your_search(index=os source=daily reporting hosts reporting=yes) | eval day=if(strftime(_time,"%Y-%m-%d")=strftime(now(),"%Y-%m-%d"),"Today","Yesterday") | stats dc(day) AS dc_day values(day) AS day BY host | where dc_day=1 AND day="Yesterday" | table host
Forgot to mention that the search is a summary Report. It creates a report every midnight and push to the summary index.
So, I have to set up alert based on past two days worth data.
the logic is using eval to give a value to a a variable (in my example "day") identifying that an event is in the day of 23 ("Yesterday") or 24 ("Today").
In this way you can understand if an host sent logs yesterday and today or not.
In the filter: with the dc option in stats command your can understand when logs are present in only one day (excluding in this way the hosts that sent logs in both the days) and values(day) AS day says that logs are of yesterday and not of today (to avoid the case case that an host is sending today but not yesterday).
Thanks for the responses.
another question. What if I want to automate this alert set up? Like I want to set up everyday alert on the delta based on comparing past two day worth of data.
save the search as an alert and schedule it once a day (after midnight) using as time frame the last two days:
Only one hint to think:
with your approach, you check the hosts that sent logs in the last two days, highlighting the missing ones, but if you have an host that didn't send logs for more than two days, you loose and don't monitor it.
I usually prefer to manage the list of hosts to monitor in a lookup, so I'm sure about the perimeter to monitor; it's more expensive because you have to manually manage this list but more sure.