topic Re: How can I cache search results to avoid data recalculation? in Splunk Search

How can I cache search results to avoid data recalculation?

RiccardoV — Tue, 14 Oct 2014 08:31:52 GMT

Hi,
I have a dashboard with parameterized search (it takes three arguments from timepicker and dropdowns) that takes long time if I have a big amount of data because of two external lookups I use.

1) Is there a way to cache / save / speed-up my search? Please note that I need realtime results too, because the user can choose real time window too, I cannot use summary index because I don't need just statistical data, but fields like

name category country timestamp

2) Is there a way to avoid the recalculation of the results when I navigate away and back to the dashboard?

thanks!

Re: How can I cache search results to avoid data recalculation?

jrodman — Tue, 14 Oct 2014 09:31:10 GMT

"Report Acceleration" is precisely storing the search-peer answers to a particular class of search, but it's quite tricky to apply this to a dashboard where the search gets parameterized. We would need to have exactly equivalent fully qualified searches to the paramaterized version of the searches in order to know that we can correctly produce the answers ahead of time and safely use them.

Another approach is to simply schedule the searches ahead of time for your dashboard, but again this involves computing the parameter-space that will be used ahead of time.

We don't have a lazy-reuse option. Typically that's not desirable because the answers for searches may change over time, so simply doing something like lazily cacheing answers to ad-hoc searches would be unhelpful.

How to most efficiently get your results is kind of particular to the goals of your searches. For example if you just want to compute reports on a relatively small set of field values , data model acceleration may be the ideal path.

Re: How can I cache search results to avoid data recalculation?

musskopf — Wed, 15 Oct 2014 02:51:04 GMT

You could use summary indexes, just use "stats" command at the end add lots of "by clause", like:

search bla | stats count by name, category, country, timestamp

But remember that at summary index will only improve the performance if the data set is smaller.

You might get better performance creating different summary indexes, one for each report or at least grouping things together where it makes sense... also, if you'll be using "by _time", consider using a | bucket span=1h _time (or any other time span) before your stats command to group things a bit.

Re: How can I cache search results to avoid data recalculation?

RiccardoV — Wed, 15 Oct 2014 08:36:29 GMT

Hi musskopf and thanks for your reply.
I think that I can't use summary index here because my search is parameterized with two user inputs (sourcetype and field to lookup). So I cannot foresee which inputs the user will use! How can I set up a summary index if I don't know the actual query in advance?

Re: How can I cache search results to avoid data recalculation?

musskopf — Wed, 15 Oct 2014 21:26:07 GMT

Sure, there are some situations where you need to know better what the report is supposed to return. In my case I do have input fields, for example, country, sex and age.

If you let the user chose any of the fields it won't help. Might be better understand what are the more common combinations to produce one fast report and another very flexible but slow. Btw, are you searching over all indexes as well? limiting the search to a specific index always help.

Re: How can I cache search results to avoid data recalculation?

RiccardoV — Thu, 16 Oct 2014 07:09:36 GMT

I'm searching on a single index and on the sourcetype that user will set on rutime

Re: How can I cache search results to avoid data recalculation?

cpt12tech — Thu, 30 Jul 2015 19:37:45 GMT

Export to a lookup table

Then run your dashboard searches from the lookup table.
http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/