topic Re: How can I real-time search in the range now-48h to now-24h? in Splunk Search

How can I real-time search in the range now-48h to now-24h?

hylam — Tue, 29 Sep 2020 07:40:09 GMT

Can I real-time search for the last 48 hours and hide the results in the last 24 hours? How about now-30d to now-29d? How about now-52wk to now-52wk+1d?

EDIT1

use case 1

Suppose I started a screen recorder 48 hours ago and stopped it 24 hours ago. During that 24 hours I was running a real-time search picking "last 24 hours". Now I don't have that recording but I want to reconstruct that real-time search animation. It works like CCTV replay showing how the burglar was breaking in.

use case 2 It is October now. I was given the historical data in September. I would like to show a splunk real-time search as the data was ingested in real-time. I need that animation.

EDIT2
I am running the following as a real-time search w/ "last 30 sec"

5 sec heartbeat

index=_internal | stats max(_time) as _time | eval _time=floor(_time/5)*5

but failed to accumulate results on the screen

| eval t0=_time-86400 | eval t1=t0+300 | map search="index=_internal starttimeu=$t0$ endtimeu=$t1$"

EDIT3
Alternative approach
https://answers.splunk.com/answers/320184/refreshing-a-dashboard-wo-grey-waiting-for-data.html

EDIT4
Case 193187 - "Replay" command
http://wiki.splunk.com/Community:ERs

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sat, 24 Oct 2015 14:20:13 GMT

You should be able to get what you need from this Q&A:

https://answers.splunk.com/answers/148842/timechart-with-time-x-axis-delineated-in-t-minutes-before-now.html

If so, add a comment with your final solution; if not, also add a comment to clarify your question.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sat, 24 Oct 2015 14:39:22 GMT

You are changing the labels on the time axis only, which is NOT what I asked for. I need the time range of the search to change in real-time.

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sat, 24 Oct 2015 14:59:37 GMT

IMHO, you have not clearly explained exactly what you need. Given this, it should not surprise you that people who are trying to help you will misunderstand you.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sat, 24 Oct 2015 15:24:56 GMT

thx. i have rephrased the question

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sat, 24 Oct 2015 15:43:00 GMT

Your desire makes little sense to me. The entire point of real-time is to have events which have "just happened" to be displayed instantly. Real-Time searches are incredibly expensive (locks a core on every server). In any case, to do the amazingly-expensive and strange thing that you said, you should be able to use this question to create a new timepicker value (I am not sure why "now" is greyed out in "latest" for "real-time" in the timepicker but it is):

https://answers.splunk.com/answers/33093/defining-a-real-time-search-window.html

Try adding this setting:

[rt-yesterdayish]
label = Real-Time Yesterday but not Today
earliest_time = rt-2d
latest_time = rt-1d
order = 10

Re: How can I real-time search in the range now-48h to now-24h?

esix_splunk — Sat, 24 Oct 2015 15:46:14 GMT

So I take your question to mean you want to search from "now" to -n Hours / Days / Months? Real time searches are typically windows for last 30 seconds / 1 minute and continually run within those windows. Anything over that should be evaluated as historical searches, not 'real-time'. However, operationally these are generally considered real-time up to about 10 minutes. But I digress.

So do you want to see this in the search bar? As a saved search? As a dashboard? Your question has a lot different variables but your not defining what you want concretely.

1) So as a dashboard..

You can create a multi-input dashboard, and define earliest.time( -1d@d, -1w@w, -72d@d) and latest.time (now, -1m@m, -1h@h) and then run your search.

2) Saved searches
Create a saved search for every time division, and mail /view results

3) Manual searches.
Run the search, use the visual browser to zoom into the desired time windows.

There are at 3 methods to accomplish similar things you are describing.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sat, 24 Oct 2015 16:04:27 GMT

Your desire makes little sense to me
I have added 2 use cases above.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sat, 24 Oct 2015 16:05:26 GMT

Please see the edit. I need that real-time search animation on historical data.

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sat, 24 Oct 2015 21:39:49 GMT

I retract my negative (and pesumptive) comments. Your use cases are very intriguing and I actually might need to do this sometime!

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sat, 24 Oct 2015 21:40:32 GMT

Did this solution not work?

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sat, 24 Oct 2015 23:58:12 GMT

Not work.

C:\opt\splunk\etc\apps\search\local\times.conf

I have added your stanza. But I have not found the new choice under the time picker under all 6 folders. I have tried http://localhost:8000/debug/refresh and restarting the server.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sun, 25 Oct 2015 00:02:16 GMT

Now you get my CCTV use case. If I get the splunk CCTV playback working at 1x speed, I should start looking for the fast motion and slow motion buttons. You know the 2x 4x 8x and the 0.5x 0.25x 0.125x speeds.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sun, 25 Oct 2015 01:07:28 GMT

Please see EDIT3 for an alternative approach.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sun, 25 Oct 2015 01:08:14 GMT

Please see EDIT3 for an alternative approach.

Re: How can I real-time search in the range now-48h to now-24h?

Richfez — Sun, 25 Oct 2015 12:57:32 GMT

I understand youe use case. I had an Enhancement Request in to Splunk for this same functionality. I'm about to close that ER as soon as I actually test and confirm that SA-Eventgen can do what I need, but it's readme indicates it should be able to.

Here's a blog post about how to use SA-Eventgen. Read through the description, note especially:

Added replay mode to allow us to replay a file from another Splunk instance to a new Splunk instance, leaking out events with proper time spaced between them to make it look like they are being generated in real time.

I have not yet actually tried it but there shouldn't be any issues doing it "from" and "to" the same Splunk instance.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sun, 25 Oct 2015 13:12:31 GMT

Thx. Let me take a look at it. I have also considered writing a python script to read historical log and do throttled output according to the timestamps. Splunk will than tail follow the throttled output. After each run I will clean eventdata. Of course I could playback at 2x 4x 8x.

One way to do it with a single splunk instance would be reading from index=history and writing to index=playback. After each playback I would clean eventdata index=playback. The real-time search dashboard should be reading from index=playback.

The "refresh dashboard panel w/o grey msg" should be useful in a number of use cases, including this one.

Re: How can I real-time search in the range now-48h to now-24h?

Richfez — Sun, 25 Oct 2015 14:04:37 GMT

Great, let us know what you find out, I think there are several interested persons following this thread now.

Re: How can I real-time search in the range now-48h to now-24h?

hylam — Sun, 25 Oct 2015 14:06:29 GMT

Can SA-eventgen do throttled playback at historical timestamps w/o replacing it at current timestamp?

Re: How can I real-time search in the range now-48h to now-24h?

woodcock — Sun, 25 Oct 2015 22:36:15 GMT

Regardless of your approach you need to be very careful about one thing: The view that you get by shifting a real-time window back in time to "watch it again" might not be the same as the view when it went by the first time. Consider the case of a 5-minute real-time window going from rt-5m to now and another from rt-10m to rt-5m. Now imagine that you have some events with latency of 7 minutes. These will show in the second window but not the first one. This kind of problem is FAR more likely to happen with shorter windows that the 1-day ones that you are investigating but it is something that you need to beware.