Hi Splunk Community, I need to build an alert that will be triggered if a specific signature is not present in the logs for a period of time. The message shows up in the logs every 3 or 4 seconds in BAU conditions, but there are some instances of longer intervals going up to 4 minutes. What I had in mind was a query that ran over a 15-time timeframe using 5-minute buckets - to ensure that I would catch the negative trend and not only the one offs. I have made it this far in the query: index=its-em-pbus3-app "Checking the receive queue for a message of size"
| bin _time span=5m aligntime=@m
| eval day_of_week = strftime(_time,"%A")
| where NOT (day_of_week="Saturday" OR day_of_week="Sunday")
| eval date_hour = strftime(_time, "%H")
| where (date_hour > 7 AND date_hour < 19)
| stats count by _time **I only need the results for Monday to Friday between the hours of 7AM and 7PM. The query returns the count by _time, which is great, but if the signature is not present I don't get any hits, obviously. So I can count the number of occurrences within the 5-minute buckets, but I can't assess the intervals or determine the absence using count. I thought of, perhaps, manipulating timestamps so I could calculate the difference between current time and the last timestamp of the event, but I am not exactly sure how to compare a timestamp to "now". I would appreciate if I could get some advice on either how to count "nulls" or how to cross-reference the timestamps of the signature against current time. Thank you all in advance.
... View more