Splunk Search

Is it better to use loadjob or scheduled saved search to improve dashboard performance?

calebwidmer
Explorer

We're writing Simple XML dashboards that utilize summary indexes for the aggregated data, but that is getting too big, and we still have to report on it.. Some of these searches' earliest is @30d@d (past 30 days), and we can't get around this. The dashboards take over two minutes to load. I looked into scheduled saved searches and loadjob, and have successfully lowered the page load time down to 5 seconds (wahoo!) by using loadjob .

Where I'm confused is, some of answers blog responses (more than one I've found) advise that using a (scheduled) saved search alone is enough to improve performance on dashboard. For me, I've found using a saved search in a dashboard has the same page loading time as the saved search definition itself.

Am I missing something?

Trying to rationalize this, maybe the Limit total jobs disk quota parameter can be hitting the ceiling and that's why the saved search is consistently rerunning the queries on the dashboard (i.e. it's capped out before the dashboard finishes), instead of previously loaded results?

What I'm looking for:
Can anyone verify that saved searches alone can/do cache search results, and is there any further tweaking that needs to be taken into consideration for large/long running searches?

Any help is appreciated.
Thank You

1 Solution

snoobzilla
Builder

Working through some of the same issues myself... question is what is latest time of your search?

If your latest time the dashboard is trying to display (e.g. now) is after the end of the scheduled save search time range (midnight last night), I believe the dashboard will recognize this for a saved search and rerun the search. Dashboards are not that "smart" around loadjob.

PS... if you are working with loadjobs you may find this post interesting on how to narrow timerange after the fact with a time picker. https://answers.splunk.com/answers/188469/how-to-get-results-to-load-with-a-time-picker-sett.html#an....

View solution in original post

snoobzilla
Builder

Working through some of the same issues myself... question is what is latest time of your search?

If your latest time the dashboard is trying to display (e.g. now) is after the end of the scheduled save search time range (midnight last night), I believe the dashboard will recognize this for a saved search and rerun the search. Dashboards are not that "smart" around loadjob.

PS... if you are working with loadjobs you may find this post interesting on how to narrow timerange after the fact with a time picker. https://answers.splunk.com/answers/188469/how-to-get-results-to-load-with-a-time-picker-sett.html#an....

calebwidmer
Explorer

Snoobzilla,
thanks for the quick response.
To rephrase, in order to utilize the cached result of a saved search on a dashboard, the visualization's earliest/latest has to within the time frame of the saved searches data, if not it will ignore the cached result and rerun the saved search query with the new earliest/latest?
FYI the job is scheduled every 10 minutes.

0 Karma

andrewtrobec
Motivator

Was reading up on this loadjob vs. savedsearch topic and would like to highlight what I found concerning savedsearch. Source is https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Savedsearch.
under usage section:

The savedsearch command always runs a new search. To reanimate the results of a previously run search, use the loadjob command.

Not sure if this was implemented later on, but the statement concerning cached results in incorrect.

gjanders
SplunkTrust
SplunkTrust

@andrewtrobec, your quoting the | savedsearch command, not to be confused with using a ref= within a dashboard which would use the last scheduled run...

snoobzilla
Builder

When I have had success with save searches, I converted to report in the web interface. I suspect that removed the time tokens for that search. Let me dig up syntax.

<panel>
  <title>Major Incident Count by Program</title>
  <table>
    <search ref="KCI_MI_COUNT_BY_PROGRAM"></search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="drilldown">none</option>
    <option name="dataOverlayMode">none</option>
    <option name="count">10</option>
  </table>
</panel>

Yes, I was correct... there is no time picker tokens, so if you have any in here that is probably why it is trying to run the search.

If you convert to report in the interface you should get syntax above.

calebwidmer
Explorer

Thank you that did it!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...