Why is Dashboard Performance So Bad?

[David]

I've got a few lengthy dashboard load times that I'm trying to improve. Short of professional services, I'd like to poll the Answers audience for performance improvement suggestions. Is a 30+ second load time just the way it has to be when using a lot of charts?

General Guidelines

Things I have implemented / am implementing / am looking at so far:

• Summary Indexing
• Using One Search for Multiple Charts
• Inspecting the actual searches to analyze performance via the Job Inspector
• Run Searches in Advance and Use Cached Results
• Evaluate Server Performance with respect to Splunk
• Configure your flashtimeline appropriately
• Pull out any remaining cruft from SimpleXML to Advanced XML Conversion (Particularly EnablePreview, if not needed)

My Specific Test Case

I have a particular view (just one example from many with this issue) that looks at a small amount of data, then displays 30+ FlashCharts with different views. It's process is:

1. Set up Chrome, Time Picker
2. Run a generic search for * | head 1 to support custom binning and indexes. (why?)
3. Execute Custom Javascript to support custom binning and indexes.
4. Execute primary search, which takes 1 second to complete based on job inspector.
5. Execute 36 PostProcesses to support flashcharts.
6. Execute 36 additional PostProcesses to support textual data (sideview HTML modules are faster than SimpleResultsTable modules)

The behavior seen by the end user is as follows:

1. Time=0 - User Clicks Refresh on dashboard
2. Time=4 - Display clears of old data, we see the Loading...
3. Time=12 - All the textual data displays (Sideview HTML Modules)
4. Time=12 - All Flash on every instance of my browsers stops functioning (including my flash-based stopwatch)
5. Time=30 - All the flash charts fill in.

That is in Chrome. In Firefox, the textual data (#3) fills in at Time=31, just after the flash charts.

This search traverses about 5k of uncompressed source data (1.2k events). The server itself has two dual core Intels (so it's a couple of years old), which is generally bored and only spikes the CPU of one core for about 2 seconds during the search. The test box is brand new (nothing fancy or unusual installed) with a Core i5 2.53 GHz processor, discrete graphics and 4 GB of ram.

Is there anything else I can do to make this better?

[David · Accepted solution]

Just syncing up for posterity. It's been 2.5 years. I just updated my app in Splunk 6 with the current version of SideView and switched all of the FlashCharts to the new JSCharts. After all of that (and on a new server), the entire page now loads in about six seconds. This is consistent with all the insights from Alex and Nick, years ago.

[David]

To try and see if there was a root cause in anything I am doing, I totally rebuilt the view in simplexml with just a search and a bunch of <chart><searchPostProcess>s. Similar results.

[David]

This is a dashboard for a particular product, showing the stats for a number of key customers. We also have a per-customer page where they can select the customer, but we're replacing another monitoring system that has this dashboard, and the business is very reliant on it. As far as performance, we're bridged by a 100 MBPS connections with 4 ms latency, so that definitely shouldn't be the issue. I'll try to replicate the slowness in a totally different scenario. If it seems like just FlashChart slowness though, is there anything that can be done to improve it?

[araitz]

I have seen FlashChart suffer the most over low-bandwidth, high latency connections (2G EVDO or so), or when using FF on an underpowered machine also doing indexing and UI. I haven't tried the kind of profiling such as you have undertaken, so it is interesting to see the results. What kind of visualizations do you need 36 of, by the way? Maybe there is a better solution.

[David]

It does seem that FlashChart is the bottleneck. I replaced all the FlashCharts with SimpleResultsTables (to minimize the delta) and it loaded up in 7 seconds. That clocks in at about .68 seconds per flash chart. Does that number make sense, or should it be able to present the objects faster?

[sideview]

I have the same question as Alex -- what happens when there are no FlashCharts? If you replace them all with SimpleResultsTable modules do things seem to return to normal?

The FlashCharts have a feature where they all sort of communicate with each other to try and share the same color space. It's possible that this communication doesnt scale well when there are that many FlashCharts on the page.

You may have seen this phenomenon already on a small scale -- you might see FlashChart A render, then FlashChart B will render, then A will sometimes adjust its colors to match B's. This is just a weird hunch but I can see this not working out super well when there are actually 30 of them.

[David]

So in the interest of testing, I re-wrote the view in simple xml, sans sideview, custombehavior, lister module, etc. The results were approximately the same: http://pastebin.com/krk7p008

[David]

I refreshed the page very close to t=0, so it looks like it's about 2.5 seconds till the first initialize, then that proceeds for 3 seconds. Afterward, the connectBridges go for 1.2 seconds, and then we wait for 8 seconds, at which part we get the onConnects and onContextChanges, which run for another 11 seconds. Then we get the results_preview calls and some other calls which take a few seconds to load, and then the page finishes.

[David]

In the .js files, neither have a onSwfReady; FlashWrapper.js has no onContextChange and FlashChart has no connectBridge.

I added the logging to the other functions, though, and here's the firebug output: http://pastebin.com/qrpXWmHP

[sideview]

And in firefox console.log will write out to the firebug console. You can use Splunk's this.logger.debug() stuff too instead of the plain old console, but you'd have to set js_logger_mode = Firebug in web.conf's [settings] stanza first.

[sideview]

No, I would just change permissions on the files in place so you can edit them. Then add this sort of thing:::::
var now = new Date();
console.log(this.moduleId + " onConnect at " + now.getSeconds() + "." + now.getMilliseconds())

[David]

Sure. What's the best way to do that? Copy the .js files into modules (a la sideview) and use this.logger.error()? And then where would I get access to the log file?

[sideview]

If you have any interest going well above and beyond the call of duty, you could add some debug to FlashChart.js and FlashWrapper.js, to print out times and moduleIDs for - initialize, onSwfReady, connectBridge, onConnect, and perhaps onContextChange. FlashChart has a very complicated load sequence. While .68 seconds per chart doesnt sound surprising, it would be interesting to know which phase was taking all the time.

[David]

So FlashCharts are indeed the pain point -- replacing them with SimpleResultsTables lowered the total time to 7 seconds. I tried to test out your theory by adding <param name="charting.legend.masterLegend"></param> to every chart (all but two of which have their colors defined by <param name="charting.fieldColors">{"FieldA": 0x8A8621}</param> anyway). I didn't get any improvement in time. Is it possible that it just takes .68 seconds per FlashChart to load?

[araitz]

Seems like FlashChart is the bottleneck, no? What happens when there are no FlashCharts?