Changing a real-time search to date range search

[matthewcanty]

Hi all. I have been working on a query to alert us to when a date changes for a specific item, the item is represented in my query as GroupId. Dates can change any number of times, and I want to receive an alert whenever they do.

I have come up with the following real-time search. I am concerned about it only having the data available from when the search starts.

index=daluat PublishedGtpRule GroupId | dedup 100 GroupId sortby -_time | stats first(Date) As CurrentDate list(Date) As PreviousDates dc(Date) As Count by GroupId | where Count>1

---

I'll try my best to explain what I think would work best.

1. Search for distinct GroupIds over a time period
2. For each GroupId, search for the previous occurrence of it - all time
3. Result if date has changed

Preferably I'd like to see what the new and previous dates are too.

This way my alerting doesn't have to be real-time. I can simply run it every minute, and look back over the last 2 or 3 minutes and allow the subsearch to go back through all time to find the last occurrence of a GroupId.

FYI the number of GroupIds per minute is a low number, 10 at most.

[matthewcanty · Accepted solution]

I have solved my problem. If anyone has any cleaner suggestions I would be more than happy to hear from you. I hope this helps someone one day.

I run the following search on a schedule every 10 minutes with a time range from -1y to now.

index=daluat Published GroupId 
| DEDUP 2 GroupId 
| STATS DC(Date) AS dist_dates FIRST(Date) AS NewDate LAST(Date) AS OldDate LATEST(_time) AS latest_time BY GroupId
| EVAL relative_time_threshold=relative_time(now(),"-10m") 
| WHERE latest_time>relative_time_threshold AND dist_dates>1 
| CONVERT CTIME(latest_time) 
| FIELDS GroupId NewDate OldDate

[matthewcanty]

It's not real time anymore. It's running once every 10 minutes.

[linu1988]

if it is a realtime then shouldn't it be -1y to rt?