topic Re: Search for events that have not happened in Getting Data In

Search for events that have not happened

sheltonr — Wed, 27 Oct 2010 01:21:34 GMT

When we 'fall back' one hour for Daylight Savings Time, I'd like to run a search that would reveal those log sources that didn't adjust correctly (their event timestamps will be one hour in the future.) What might the syntax for such a search look like? Thank you!

Re: Search for events that have not happened

sideview — Wed, 27 Oct 2010 01:25:58 GMT

The way I've usually seen this done is to compare _time to _indextime.

_time is the time extracted from the event's text, as the number of seconds since 1970

_indextime is the time the event was indexed, according to the local system clock (also number of seconds since 1970).

so if you use eval to look for this difference you should have what you need.

Re: Search for events that have not happened

araitz — Wed, 27 Oct 2010 23:43:35 GMT

This assumes that your Splunk servers' clocks are affected by DST 🙂

Re: Search for events that have not happened

araitz — Wed, 27 Oct 2010 23:45:02 GMT

Ah, the nefarious "future event".

For a lightweight approach, how about:

| metadata type=hosts | eval now = now() | where latestTime > now

Re: Search for events that have not happened

sideview — Thu, 28 Oct 2010 05:04:25 GMT

Well more generally you want to see anywhere where there's a delta between _indexTime and _time If both sides respect DST there's no issue. If only one respects DST there'll be a persistent delta suddenly, and if they're slightly off there'll be a delta for some small slice of time and then it'll go away. And in each case which direction the delta goes depends on which side is observing DST and which is not.

Re: Search for events that have not happened

cudgel — Thu, 04 Nov 2010 21:22:39 GMT

How well does the search "| metadata type=hosts | eval now = now() | where latestTime > now" work in a scheduled search? I seem to get a huge load of results for hosts off by as much as 2-3 minutes because of what - now() evaluates to when the search is scheduled to run? If I run the search manually, e.g. over the past hour, it seems to return accurate results.