It is a very simple search for a string. (Account lock outs to be precise) and as worked in the past. But just recently stopped working. (We upgraded to 6.1.1 yesterday incidentally)
The event is there. I can see it if I search for other terms, and I can see it if I search "Today" but when I search "All time (real-time)" it does not come up.
This is some weird behavior. I am not sure how to investigate it from here. I would really like to know why results are not being returned.
More info
I ran across this documentation about real time searches:
"Real-time search matches events that have arrived at the port but have not been persisted to disk."
This however does not seem to square with the results from a real time search of index=_internal | stats earliest(_time) as earliest | eval earliest=strftime(earliest, "%T")
When running this search the earliest is aways a time near now minus the time range of the real time search. Obviously these results have been written to the disk, but are also returned in a real time search.
Am I misunderstanding the documentation?
An all-time real-time search is different from a real-time search with a time-frame defined in that it does not back-fill. Instead, an all-time real-time search will show events that occur after you started the search.
Try defining a time-range for your real-time search and it should begin matching events.