Can I real-time search for the last 48 hours and hide the results in the last 24 hours? How about now-30d to now-29d? How about now-52wk to now-52wk+1d?
Suppose I started a screen recorder 48 hours ago and stopped it 24 hours ago. During that 24 hours I was running a real-time search picking "last 24 hours". Now I don't have that recording but I want to reconstruct that real-time search animation. It works like CCTV replay showing how the burglar was breaking in.
I am running the following as a real-time search w/ "last 30 sec"
index=internal | stats max(time) as time | eval _time=floor(time/5)*5
| eval t0=time-86400 | eval t1=t0+300 | map search="index=internal starttimeu=$t0$ endtimeu=$t1$"
Case 193187 - "Replay" command
Regardless of your approach you need to be very careful about one thing: The view that you get by shifting a real-time window back in time to "watch it again" might not be the same as the view when it went by the first time. Consider the case of a 5-minute real-time window going from
now and another from
rt-5m. Now imagine that you have some events with latency of 7 minutes. These will show in the second window but not the first one. This kind of problem is FAR more likely to happen with shorter windows that the 1-day ones that you are investigating but it is something that you need to beware.
I understand youe use case. I had an Enhancement Request in to Splunk for this same functionality. I'm about to close that ER as soon as I actually test and confirm that SA-Eventgen can do what I need, but it's readme indicates it should be able to.
I have not yet actually tried it but there shouldn't be any issues doing it "from" and "to" the same Splunk instance.
Thx. Let me take a look at it. I have also considered writing a python script to read historical log and do throttled output according to the timestamps. Splunk will than tail follow the throttled output. After each run I will clean eventdata. Of course I could playback at 2x 4x 8x.
One way to do it with a single splunk instance would be reading from index=history and writing to index=playback. After each playback I would clean eventdata index=playback. The real-time search dashboard should be reading from index=playback.
The "refresh dashboard panel w/o grey msg" should be useful in a number of use cases, including this one.
I'm not sure, hopefully the maintainers of SA-Eventgen will notice this and answer. You could gently ask them if you'd like, they may be able to give some additional insight into this.
If I had to hazard a guess about if it could do the replay without replacing the timestamps, I would think it might make the quantum superposition of our universe end and cause our reality to collapse into some Salvador Dali like painting where our heads all melt like those clocks. I'd not recommend it.
So I take your question to mean you want to search from "now" to -n Hours / Days / Months? Real time searches are typically windows for last 30 seconds / 1 minute and continually run within those windows. Anything over that should be evaluated as historical searches, not 'real-time'. However, operationally these are generally considered real-time up to about 10 minutes. But I digress.
So do you want to see this in the search bar? As a saved search? As a dashboard? Your question has a lot different variables but your not defining what you want concretely.
1) So as a dashboard..
You can create a multi-input dashboard, and define earliest.time( -1d@d, -1w@w, -72d@d) and latest.time (now, -1m@m, -1h@h) and then run your search.
2) Saved searches
Create a saved search for every time division, and mail /view results
3) Manual searches.
Run the search, use the visual browser to zoom into the desired time windows.
There are at 3 methods to accomplish similar things you are describing.
Your desire makes little sense to me. The entire point of real-time is to have events which have "just happened" to be displayed instantly. Real-Time searches are incredibly expensive (locks a core on every server). In any case, to do the amazingly-expensive and strange thing that you said, you should be able to use this question to create a new timepicker value (I am not sure why "now" is greyed out in "latest" for "real-time" in the timepicker but it is):
Try adding this setting:
[rt-yesterdayish] label = Real-Time Yesterday but not Today earliest_time = rt-2d latest_time = rt-1d order = 10
I have added your stanza. But I have not found the new choice under the time picker under all 6 folders. I have tried http://localhost:8000/debug/refresh and restarting the server.
Now you get my CCTV use case. If I get the splunk CCTV playback working at 1x speed, I should start looking for the fast motion and slow motion buttons. You know the 2x 4x 8x and the 0.5x 0.25x 0.125x speeds.
You should be able to get what you need from this Q&A:
If so, add a comment with your final solution; if not, also add a comment to clarify your question.