I use alert manager datamodel to keep track of all the invoked alerts month over month. Using the following: (index=* OR index=_*) (eventtype="alert_metadata") | where label != "NULL" |chart count over label by Time | rename label AS Alert | addcoltotals labelfield=Alert label=TOTAL It was working great. I was able to see the trends of how many time the alerts were triggered each month for the past 3 or 5 months. Then, the numbers that I saw last month does not match with what I see this month. For example: When the spl above ran on Aug. 3rd, it showed the the number of alerts fired for July is 171 When the spl above ran on Sept. 3rd, it showed the the number of alerts fired for July is 145 ... missing 26 When the spl above ran on Sept. 9, it showed the the number of alerts fired for July is 123 ... missing 22 It seems the alerts keep disappearing from the alert manager data model. Has any body seen this type of behavior before? Is there any suggestion to fix this problem? Thanks.
... View more