I understand that indexed real-time searches are not up to the second and lag (default of 60 sec) behind a real-time search. With this being the case, what is the difference between these two searches?
Scheduled search:
savesearches.conf
cron_schedule = * * * * *
dispatch.earliest_time = -1m
dispatch.latest_time = now
AND
Indexed real-time search **
**limits.conf
indexed_realtime_disk_sync_delay = 60
Realtime takes much more ressources and I disable it by default. Ad-hoc search let your search into the stored data in the index tier. Realtime search send you the events before storing them. He have around 150 servers for Splunk at 40 000$ each and we still a reluctant to give that capability since we prefer to waste our CPU elsewhere.
One key difference is that your scheduled search will inevitably miss events. A practical example:
23:45:59 event happens
23:46:00 search 1 runs for 23:45-23:46
23:46:01 event is indexed
23:47:00 search 2 runs for 23:46-23:47
Search 1 doesn't have the event yet, search 2 doesn't care about the event any more.
If you know your delays you could build them into your time range, or even better run on indexed time for your main interval timerange.
great example. thanks again @martin_mueller