A search that does not produce results for "All time (real-time)" but does for "Today"

[neiljpeterson]

It is a very simple search for a string. (Account lock outs to be precise) and as worked in the past. But just recently stopped working. (We upgraded to 6.1.1 yesterday incidentally)

The event is there. I can see it if I search for other terms, and I can see it if I search "Today" but when I search "All time (real-time)" it does not come up.

This is some weird behavior. I am not sure how to investigate it from here. I would really like to know why results are not being returned.

---

More info

I ran across this documentation about real time searches:

Documentation > Splunk Enterprise > Search Manual > Expected performance and known limitations of re...

"Real-time search matches events that have arrived at the port but have not been persisted to disk."

This however does not seem to square with the results from a real time search of index=_internal | stats earliest(_time) as earliest | eval earliest=strftime(earliest, "%T") When running this search the earliest is aways a time near now minus the time range of the real time search. Obviously these results have been written to the disk, but are also returned in a real time search.

Am I misunderstanding the documentation?

[LukeMurphey]

An all-time real-time search is different from a real-time search with a time-frame defined in that it does not back-fill. Instead, an all-time real-time search will show events that occur after you started the search.

Try defining a time-range for your real-time search and it should begin matching events.