Search for events that have not happened

[sheltonr]

When we 'fall back' one hour for Daylight Savings Time, I'd like to run a search that would reveal those log sources that didn't adjust correctly (their event timestamps will be one hour in the future.) What might the syntax for such a search look like? Thank you!

[sideview · Accepted solution]

The way I've usually seen this done is to compare _time to _indextime.

_time is the time extracted from the event's text, as the number of seconds since 1970

_indextime is the time the event was indexed, according to the local system clock (also number of seconds since 1970).

so if you use eval to look for this difference you should have what you need.

[cudgel]

How well does the search "| metadata type=hosts | eval now = now() | where latestTime > now" work in a scheduled search? I seem to get a huge load of results for hosts off by as much as 2-3 minutes because of what - now() evaluates to when the search is scheduled to run? If I run the search manually, e.g. over the past hour, it seems to return accurate results.

[araitz · Accepted solution]

Ah, the nefarious "future event".

For a lightweight approach, how about:

| metadata type=hosts | eval now = now() | where latestTime > now

[sideview · Accepted solution]

The way I've usually seen this done is to compare _time to _indextime.

_time is the time extracted from the event's text, as the number of seconds since 1970

_indextime is the time the event was indexed, according to the local system clock (also number of seconds since 1970).

so if you use eval to look for this difference you should have what you need.

[sideview]

Well more generally you want to see anywhere where there's a delta between _indexTime and _time If both sides respect DST there's no issue. If only one respects DST there'll be a persistent delta suddenly, and if they're slightly off there'll be a delta for some small slice of time and then it'll go away. And in each case which direction the delta goes depends on which side is observing DST and which is not.

[araitz]

This assumes that your Splunk servers' clocks are affected by DST 🙂