How can I real-time search in the range now-48h to now-24h?

[hylam]

Can I real-time search for the last 48 hours and hide the results in the last 24 hours? How about now-30d to now-29d? How about now-52wk to now-52wk+1d?

EDIT1

• use case 1

Suppose I started a screen recorder 48 hours ago and stopped it 24 hours ago. During that 24 hours I was running a real-time search picking "last 24 hours". Now I don't have that recording but I want to reconstruct that real-time search animation. It works like CCTV replay showing how the burglar was breaking in.

• use case 2 It is October now. I was given the historical data in September. I would like to show a splunk real-time search as the data was ingested in real-time. I need that animation.

EDIT2
I am running the following as a real-time search w/ "last 30 sec"

5 sec heartbeat

index=_internal | stats max(_time) as _time | eval _time=floor(_time/5)*5

but failed to accumulate results on the screen

| eval t0=_time-86400 | eval t1=t0+300 | map search="index=_internal starttimeu=$t0$ endtimeu=$t1$"

EDIT3
Alternative approach
https://answers.splunk.com/answers/320184/refreshing-a-dashboard-wo-grey-waiting-for-data.html

EDIT4
Case 193187 - "Replay" command
http://wiki.splunk.com/Community:ERs

[woodcock]

Regardless of your approach you need to be very careful about one thing: The view that you get by shifting a real-time window back in time to "watch it again" might not be the same as the view when it went by the first time. Consider the case of a 5-minute real-time window going from rt-5m to now and another from rt-10m to rt-5m. Now imagine that you have some events with latency of 7 minutes. These will show in the second window but not the first one. This kind of problem is FAR more likely to happen with shorter windows that the 1-day ones that you are investigating but it is something that you need to beware.

[hylam]

Can I apply the time window to _indextime instead of _time?

[Richfez]

I understand youe use case. I had an Enhancement Request in to Splunk for this same functionality. I'm about to close that ER as soon as I actually test and confirm that SA-Eventgen can do what I need, but it's readme indicates it should be able to.

Here's a blog post about how to use SA-Eventgen. Read through the description, note especially:

• Added replay mode to allow us to replay a file from another Splunk instance to a new Splunk instance, leaking out events with proper time spaced between them to make it look like they are being generated in real time.

I have not yet actually tried it but there shouldn't be any issues doing it "from" and "to" the same Splunk instance.

[hylam]

Thx. Let me take a look at it. I have also considered writing a python script to read historical log and do throttled output according to the timestamps. Splunk will than tail follow the throttled output. After each run I will clean eventdata. Of course I could playback at 2x 4x 8x.

One way to do it with a single splunk instance would be reading from index=history and writing to index=playback. After each playback I would clean eventdata index=playback. The real-time search dashboard should be reading from index=playback.

The "refresh dashboard panel w/o grey msg" should be useful in a number of use cases, including this one.

[Richfez]

Great, let us know what you find out, I think there are several interested persons following this thread now.

[hylam]

Can SA-eventgen do throttled playback at historical timestamps w/o replacing it at current timestamp?

[Richfez]

I'm not sure, hopefully the maintainers of SA-Eventgen will notice this and answer. You could gently ask them if you'd like, they may be able to give some additional insight into this.

If I had to hazard a guess about if it could do the replay without replacing the timestamps, I would think it might make the quantum superposition of our universe end and cause our reality to collapse into some Salvador Dali like painting where our heads all melt like those clocks. I'd not recommend it.

[esix_splunk]

So I take your question to mean you want to search from "now" to -n Hours / Days / Months? Real time searches are typically windows for last 30 seconds / 1 minute and continually run within those windows. Anything over that should be evaluated as historical searches, not 'real-time'. However, operationally these are generally considered real-time up to about 10 minutes. But I digress.

So do you want to see this in the search bar? As a saved search? As a dashboard? Your question has a lot different variables but your not defining what you want concretely.

1) So as a dashboard..

You can create a multi-input dashboard, and define earliest.time( -1d@d, -1w@w, -72d@d) and latest.time (now, -1m@m, -1h@h) and then run your search.

2) Saved searches
Create a saved search for every time division, and mail /view results

3) Manual searches.
Run the search, use the visual browser to zoom into the desired time windows.

There are at 3 methods to accomplish similar things you are describing.

[hylam]

Please see EDIT3 for an alternative approach.

[hylam]

Please see the edit. I need that real-time search animation on historical data.

[woodcock]

Your desire makes little sense to me. The entire point of real-time is to have events which have "just happened" to be displayed instantly. Real-Time searches are incredibly expensive (locks a core on every server). In any case, to do the amazingly-expensive and strange thing that you said, you should be able to use this question to create a new timepicker value (I am not sure why "now" is greyed out in "latest" for "real-time" in the timepicker but it is):

https://answers.splunk.com/answers/33093/defining-a-real-time-search-window.html

Try adding this setting:

[rt-yesterdayish]
label = Real-Time Yesterday but not Today
earliest_time = rt-2d
latest_time = rt-1d
order = 10

[hylam]

Please see EDIT3 for an alternative approach.

[woodcock]

Did this solution not work?

[hylam]

Not work.

C:\opt\splunk\etc\apps\search\local\times.conf

I have added your stanza. But I have not found the new choice under the time picker under all 6 folders. I have tried http://localhost:8000/debug/refresh and restarting the server.

[hylam]

Your desire makes little sense to me
I have added 2 use cases above.

[woodcock]

I retract my negative (and pesumptive) comments. Your use cases are very intriguing and I actually might need to do this sometime!

[hylam]

Now you get my CCTV use case. If I get the splunk CCTV playback working at 1x speed, I should start looking for the fast motion and slow motion buttons. You know the 2x 4x 8x and the 0.5x 0.25x 0.125x speeds.

[woodcock]

You should be able to get what you need from this Q&A:

https://answers.splunk.com/answers/148842/timechart-with-time-x-axis-delineated-in-t-minutes-before-...

If so, add a comment with your final solution; if not, also add a comment to clarify your question.

[hylam]

You are changing the labels on the time axis only, which is NOT what I asked for. I need the time range of the search to change in real-time.

[woodcock]

IMHO, you have not clearly explained exactly what you need. Given this, it should not surprise you that people who are trying to help you will misunderstand you.