topic Re: Realtime searches (efficiency & results) in Splunk Search

Realtime searches (efficiency & results)

troywollenslege — Fri, 09 Nov 2012 15:27:21 GMT

As far as efficiency, we were told that realtime searches take "a fraction" of a CPU core per search. Does it matter if someone is doing realtime-all time, or realtime-5min/30 min window?

I was looking at http://www.splunk.com/view/real-time-in-splunk/SP-CAAAFD7 but wasn’t clear.

Just confirming, if you do a rt30 minute search and you look for an event that follows another event and those events are >30 minutes apart, you wouldn’t see anything.

Also, if you do realtime search for 30 minute window and events come into Splunk with different time greater than 30 minutes (e.g. timezone or bad CPU clock time) you won't see those as either.

Thanks

Re: Realtime searches (efficiency & results)

phoenixdigital — Wed, 10 Apr 2013 00:36:59 GMT

I would like to know the answer to this as well if anyone knows.

Re: Realtime searches (efficiency & results)

dglinder — Fri, 08 Nov 2013 17:03:08 GMT

bump I hate adding a "me too" for a response...

Re: Realtime searches (efficiency & results)

iTechEvent — Mon, 07 Apr 2014 00:04:29 GMT

Windowed Real Time: uses earliesttime=rt-1 latesttime=rt
Non Windowed Real Time: earliesttime=rt latesttime=rt

Let me share my experience so far on this matter:
They can be compared with these parameters:
1.cpu 2.ram 3. diskspace 4. #events 5. querytime 6. overhead

Windowed RT
cpu-fraction of core per search as above
events-Query returns mostly fixed number of events, with some marginal fluctuation
querytime-mostly the same due to event count
ram-mostly fixed amount with some fluctuations due to above events count
diskspace for query-increasing and can exhaust the disk space quota per user
overheads-Window management overhead

Non Window RT
cpu-fraction of core per search as above
events-Query returns continuously increasing number of events since its all real time and the events continue to increase over time
querytime- as the event counts increases, the query run time also increase due to processing more events
ram-increasing amount of ram consumed as the event count keeps increasing
diskspace for query- same as windowed,increasing and can exhaust the disk space quota per user
overheads-No window management overhead, uses all events in real time

In comparison, Windowed RT preferable even though there is rolling window management overheads due to above plus points. The only minus point is the disk space keeps on increasing and can exhaust the quota. Hence periodically the windowed real time query can be disabled and enabled to clean up the disk space used and start over.