My management (and me as well, of course) loves the way the visualizations for real time searches look. But from a system administration perspective, it's a nightmare, as we are all well aware of the impact real time searches have on the system.
To clarify, a real time search, when it updates its data, seamlessly and continuously transforms the current data point to the next data point as new data streams through Splunk. However, if we use a historical search with an auto-refresh, there is sort of 'flash' of blank space as the search runs and populates the visualization with the newly retrieved data.
I am looking for a solution to replace the real time searches with a historical search without impacting the visualization - in other words, a historical search that displays like a real time search (only, of course without the continual updates to the values).
Is this possible, has any one tried this or have any ideas? Again, the idea is to give management the 'smooth' visualizations they have come to expect from the real time searches without having to actually run a real time search.
The number of concurrent real-time searches can greatly affect indexing performance. To lessen the impact on the indexer, you can enable indexed real-time search. This runs the search like a historical search, but also continually updates it with new events as they appear on disk. To enable indexed real-time search as the default behavior for your real-time searches, edit the limits.conf stanza called realtime and set indexed_realtime_use_by_default = true. Indexed real-time search is used when up-to-the-second accuracy is not needed. The results returned by indexed real-time search will always lag behind a real-time search. You can control the number of seconds of lag with the indexed_realtime_disk_sync_delay = setting. By default, this delay is 60 seconds.