Dashboards & Visualizations
Highlighted

Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

I've got a few lengthy dashboard load times that I'm trying to improve. Short of professional services, I'd like to poll the Answers audience for performance improvement suggestions. Is a 30+ second load time just the way it has to be when using a lot of charts?

General Guidelines

Things I have implemented / am implementing / am looking at so far:

My Specific Test Case

I have a particular view (just one example from many with this issue) that looks at a small amount of data, then displays 30+ FlashCharts with different views. It's process is:

  1. Set up Chrome, Time Picker
  2. Run a generic search for * | head 1 to support custom binning and indexes. (why?)
  3. Execute Custom Javascript to support custom binning and indexes.
  4. Execute primary search, which takes 1 second to complete based on job inspector.
  5. Execute 36 PostProcesses to support flashcharts.
  6. Execute 36 additional PostProcesses to support textual data (sideview HTML modules are faster than SimpleResultsTable modules)

The behavior seen by the end user is as follows:

  1. Time=0 - User Clicks Refresh on dashboard
  2. Time=4 - Display clears of old data, we see the Loading...
  3. Time=12 - All the textual data displays (Sideview HTML Modules)
  4. Time=12 - All Flash on every instance of my browsers stops functioning (including my flash-based stopwatch)
  5. Time=30 - All the flash charts fill in.

That is in Chrome. In Firefox, the textual data (#3) fills in at Time=31, just after the flash charts.

This search traverses about 5k of uncompressed source data (1.2k events). The server itself has two dual core Intels (so it's a couple of years old), which is generally bored and only spikes the CPU of one core for about 2 seconds during the search. The test box is brand new (nothing fancy or unusual installed) with a Core i5 2.53 GHz processor, discrete graphics and 4 GB of ram.

Is there anything else I can do to make this better?

Tags (2)
Highlighted

Re: Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

Seems like FlashChart is the bottleneck, no? What happens when there are no FlashCharts?

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

SplunkTrust
SplunkTrust

I have the same question as Alex -- what happens when there are no FlashCharts? If you replace them all with SimpleResultsTable modules do things seem to return to normal?

The FlashCharts have a feature where they all sort of communicate with each other to try and share the same color space. It's possible that this communication doesnt scale well when there are that many FlashCharts on the page.

You may have seen this phenomenon already on a small scale -- you might see FlashChart A render, then FlashChart B will render, then A will sometimes adjust its colors to match B's. This is just a weird hunch but I can see this not working out super well when there are actually 30 of them.

Highlighted

Re: Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

So FlashCharts are indeed the pain point -- replacing them with SimpleResultsTables lowered the total time to 7 seconds. I tried to test out your theory by adding <param name="charting.legend.masterLegend"></param> to every chart (all but two of which have their colors defined by <param name="charting.fieldColors">{"FieldA": 0x8A8621}</param> anyway). I didn't get any improvement in time. Is it possible that it just takes .68 seconds per FlashChart to load?

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

SplunkTrust
SplunkTrust

If you have any interest going well above and beyond the call of duty, you could add some debug to FlashChart.js and FlashWrapper.js, to print out times and moduleIDs for - initialize, onSwfReady, connectBridge, onConnect, and perhaps onContextChange. FlashChart has a very complicated load sequence. While .68 seconds per chart doesnt sound surprising, it would be interesting to know which phase was taking all the time.

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

Sure. What's the best way to do that? Copy the .js files into modules (a la sideview) and use this.logger.error()? And then where would I get access to the log file?

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

SplunkTrust
SplunkTrust

No, I would just change permissions on the files in place so you can edit them. Then add this sort of thing:::::
var now = new Date();
console.log(this.moduleId + " onConnect at " + now.getSeconds() + "." + now.getMilliseconds())

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

SplunkTrust
SplunkTrust

And in firefox console.log will write out to the firebug console. You can use Splunk's this.logger.debug() stuff too instead of the plain old console, but you'd have to set jsloggermode = Firebug in web.conf's [settings] stanza first.

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

In the .js files, neither have a onSwfReady; FlashWrapper.js has no onContextChange and FlashChart has no connectBridge.

I added the logging to the other functions, though, and here's the firebug output: http://pastebin.com/qrpXWmHP

0 Karma
Highlighted

Re: Why is Dashboard Performance So Bad?

Splunk Employee
Splunk Employee

I refreshed the page very close to t=0, so it looks like it's about 2.5 seconds till the first initialize, then that proceeds for 3 seconds. Afterward, the connectBridges go for 1.2 seconds, and then we wait for 8 seconds, at which part we get the onConnects and onContextChanges, which run for another 11 seconds. Then we get the results_preview calls and some other calls which take a few seconds to load, and then the page finishes.

0 Karma