I solve this by doing two things:
In props.conf:
[source:://my/watched/path/*]
CHECK_METHOD = modtime
My events don't have time stamps, so _time is now set to the last time the file was touched.
In my searches:
<base search> | eventstats max(_time) as LatestTime by source
| eval ThreshTime = LatestTime - 2
| where _time > ThreshTime
| <now the rest of my search>
Find the latest _time, subtract a few seconds to account for the time it might take to ingest (might need more here if coming through a forwarder?) and ignore events that aren't in this window. I combine this with a time range spec that goes back a few days so that the initial search doesn't have to work as hard.
I have to think there's a better way to do this, but I have not found it. I too am looking to present a dashboard of data based on a file that's refreshed daily, but the update happens at varying times of day, or sometimes not at all, so just doing "earliest=-1d@d" doesn't work. If the boss logs in before the refresh happens, the dashboard would be blank. Doing "earliest=-24h" might catch two days' worth of data if done at the wrong time, so that's out too.
I'm not super happy with it because it makes all my searches more expensive, but it gets the job done.
I went down a path of trying to do a scheduled search to take my code from above and reverse the sense on it:
... eval ThreshTime = LatestTime - 2 | where _time < ThreshTime | delete
But this fails because apparently delete can't be used after a streaming command like eventstats .
Any other thoughts on this I'd love to hear.
... View more
