I check the last timestamp on the newest entry in a summary index that has the same marker - I tag every entry with something related to the search populating the data so I can use a summary index for multiple results and easily sort them out. I've checked for gaps manually in the past, but I plan on automating this in some way so that when I launch a job to populate the summary index, it will figure out for itself what timestamp to start the search in order to backfill correctly. This would mean no gaps, in theory, so even if I run the job daily but it doesn't get run for days for some odd reason, I still get all my data complete in the summary index.
I'm experimenting with something like this:
sourcetype=mylogs [search index=summary_mylogs | head 1 | eval earliest=_time | return earliest] latest=now | timechart span=1m count
This subsearch gets the timestamp of the last event in the summary index and outputs earliest=### into the main search, which then uses it as the earliest start time for the search.
So far it seems to work...
... View more