Real time search suddenly incorrect

[DFresh4130]

I have a dashboard with a few radial gauges doing real time searches over the past 1 minute. They're just going over the apache logs and giving me an idea of how many hits we're getting in real time. This is one example search:

host=server1 OR host=server2 OR host=server3 OR host=server4 OR host=server5 OR host=server6 OR host=server7 OR host=server8 OR host=server9 source=www.abc.com OR source=www.123.com | stats count

Last week it was giving me numbers of over 3K and updating with no issues. Today it's only going between 50-100 when it updates. The raw search in real time where I'm able to see all the detailed entries updates says stuff like "67 of 4211 events matched" at the top just below the search bar. Looks like it's only able to keep up with the a few of the events. If I change my search to "All time (real-time)" I'm seeing the data flow in at the normal expected rate. Any suggestions on what I should be looking at?

[Unhacker]

Huh, I had a symptom like this once, and I discovered an admin had changed the local time zone setting on their server - events I received appeared to come from the future! Tellingly, I found all events when searching "All Time", but not in real- or relative-time. When a search for "(...) earliest=-5m latest=+5h (...)", returned results, the conclusion was obvious: I was receiving events from a time-traveling server, or someone had messed-up a time zone somewhere.

Maybe that helps? Check out time zones at your event sources?

Good luck, time traveler! 😉

[grijhwani · Accepted solution]

It looks like your indexer clock is wrong (or your log source time). The fact that opening it up to "all time" shows you your results is a dead giveaway.

Have you checked the actual source data for accurate timestamps? (Just lose the stats command and look at the output search results).

To prove the point you could progressively extend the real time window (using the custom search times) until your data starts to appear. (Bear in mind that the clocks could be forward of now, not necessarily behind.)

[grijhwani]

You're bettter off syncing the ESX to a standard clock and relying on VMWare to supply the time correctly.

[DFresh4130]

Yep, issue was my apache servers were about a minute behind my splunk server/indexer. Really wish NTP could be more reliable on VMware.