Dashboard refresh choking search dispatch queue

[bruceclarke]

Hey all,

I have a dashboard that is driven by a bunch of saved searches. The dashboard refreshes each panel every 30 seconds or so. As a result, I see a huge number of searches showing up in my dispatch queue and slowing down other front end searches.

Can someone point me to documentation on how to make sure these saved searches aren't choking my dispatch queue? Basically, I want to drop the search from the dispatch within a minute of it running, since I know the dashboard will refresh by then.

Thanks!

EDIT - I looked at the saved search again. It looks like whoever set this up used "All time" as the time range, but manually input earliest=-5min latest=now in the search. Would this affect how long Splunk keeps the search in the dispatch queue?

[jeffland]

Regarding the two different settings of the time range picker and the explicit earliest and latest, selecting "All Time" in the time range picker should not make a difference as long as there are not subsearches involved - these still use the time specified in the time range picker (see here).

[dolivasoh]

You can set the expiry of the saved search under Settings > Searches reports and alerts. Click the search you want to edit, scroll down to expiration and set a custom time.

[bruceclarke]

Right, but it's not a scheduled search, just a saved one. However, the saved search is used in a dashboard which the engineering team has displayed on a monitor in their room at all times. The dashboard updates each panel every 30 seconds, which appears to be what is causing the dispatch queue to grow so large.

[dolivasoh]

The same applies even for ad-hoc searches. You can configure a global default ttl for all searches if needed in your limits.conf under [search]

ttl =
* How long search artifacts should be stored on disk once completed, in seconds. The ttl is computed
* relative to the modtime of status.csv of the job if such file exists or the modtime of the search
* job's artifact directory. If a job is being actively viewed in the Splunk UI then the modtime of
* status.csv is constantly updated such that the reaper does not remove the job from underneath.
* Defaults to 600, which is equivalent to 10 minutes.

default_save_ttl =
* How long the ttl for a search artifact should be extended in response to the save control action, in second. 0 = indefinitely.
* Defaults to 604800 (1 week)

remote_ttl =
* How long artifacts from searches run in behalf of a search head should be stored on the indexer
after completion, in seconds.
* Defaults to 600 (10 minutes)

[bruceclarke]

Ok. Is there a way to edit this for just specific saved searches? Maybe in a *.meta file or config file? I don't see an option in Settings > Searches reports and alerts. I don't want to necessarily change the default for all searches.