Is it better to use loadjob or scheduled saved search to improve dashboard performance?

[calebwidmer]

We're writing Simple XML dashboards that utilize summary indexes for the aggregated data, but that is getting too big, and we still have to report on it.. Some of these searches' earliest is @30d@d (past 30 days), and we can't get around this. The dashboards take over two minutes to load. I looked into scheduled saved searches and loadjob, and have successfully lowered the page load time down to 5 seconds (wahoo!) by using loadjob .

Where I'm confused is, some of answers blog responses (more than one I've found) advise that using a (scheduled) saved search alone is enough to improve performance on dashboard. For me, I've found using a saved search in a dashboard has the same page loading time as the saved search definition itself.

Am I missing something?

Trying to rationalize this, maybe the Limit total jobs disk quota parameter can be hitting the ceiling and that's why the saved search is consistently rerunning the queries on the dashboard (i.e. it's capped out before the dashboard finishes), instead of previously loaded results?

What I'm looking for:
Can anyone verify that saved searches alone can/do cache search results, and is there any further tweaking that needs to be taken into consideration for large/long running searches?

Any help is appreciated.
Thank You

[snoobzilla · Accepted solution]

Working through some of the same issues myself... question is what is latest time of your search?

If your latest time the dashboard is trying to display (e.g. now) is after the end of the scheduled save search time range (midnight last night), I believe the dashboard will recognize this for a saved search and rerun the search. Dashboards are not that "smart" around loadjob.

PS... if you are working with loadjobs you may find this post interesting on how to narrow timerange after the fact with a time picker. https://answers.splunk.com/answers/188469/how-to-get-results-to-load-with-a-time-picker-sett.html#an....

[calebwidmer]

Snoobzilla,
thanks for the quick response.
To rephrase, in order to utilize the cached result of a saved search on a dashboard, the visualization's earliest/latest has to within the time frame of the saved searches data, if not it will ignore the cached result and rerun the saved search query with the new earliest/latest?
FYI the job is scheduled every 10 minutes.

[andrewtrobec]

Was reading up on this loadjob vs. savedsearch topic and would like to highlight what I found concerning savedsearch. Source is https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Savedsearch.
under usage section:

The savedsearch command always runs a new search. To reanimate the results of a previously run search, use the loadjob command.

Not sure if this was implemented later on, but the statement concerning cached results in incorrect.

[gjanders]

@andrewtrobec, your quoting the | savedsearch command, not to be confused with using a ref= within a dashboard which would use the last scheduled run...

[snoobzilla]

When I have had success with save searches, I converted to report in the web interface. I suspect that removed the time tokens for that search. Let me dig up syntax.

<panel>
  <title>Major Incident Count by Program</title>
  <table>
    <search ref="KCI_MI_COUNT_BY_PROGRAM"></search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="drilldown">none</option>
    <option name="dataOverlayMode">none</option>
    <option name="count">10</option>
  </table>
</panel>

Yes, I was correct... there is no time picker tokens, so if you have any in here that is probably why it is trying to run the search.

If you convert to report in the interface you should get syntax above.

[calebwidmer]

Thank you that did it!