Deployment Architecture

Splunk Web GUI Slow (search performance is fine, however)

justinbarta
Explorer

The GUI is sooooo slow. If you run a search results are pulled in fast, the page is already loaded. The problem is when navigating around the GUI loading new pages, dashboards or even configuration settings. There is a significant delay in the server sending the html page back to the user.

Our old search head which is still online 6.3.2 is blazin fast. I'm not talking about searches I'm talking about page loads. Purely the hosting of the GUI.

We built a new search head cluster on new systems which are physical, SSD loaded, tons of RAM awesomeness.
Currently these systems are 6.5.0.
Even our stand alone non-clustered 6.5.0 search heads are slow.

Does anyone have an idea to this issue?
Does anyone have an idea where the web-hosting component of splunk is and how to adjust performance settings.
Else is there a way to decouple the hosting of the splunk GUI and move it to Apache etc?

koshyk
Super Champion

hi,
Any chance to get bit more information
- which Browser you use?
- If you have Chrome -> Developer tools -> Network. Please check if any javascripts are blocked by your firewalls/proxies
- Can you please provide an output of Job Inspector (dispatch.fetch , dispatch.stream.remote, command.search.rawdata)

0 Karma

seegeekrun
Path Finder

Version 64.0.3282.186 (Official Build) (64-bit)

Nothing is blocked. The waterfall is clean, so to speak, everything loads.

No Job Inspector since this is the initial load of the Search app, i.e. no search has even run. Which is why, I can see that the issue is sitting on the server (search head) side. The time to first byte is how long the browser is waiting for Splunk (splunkweb process) to process the and sent back the response.

So the question is, from a web framework point of view, what does Splunk do on the initial load. It would seem that there's zero caching of UI elements and it's generating everything each time.

0 Karma

seegeekrun
Path Finder

As an update, I'm going to try max_view_cache_size in web.conf. We have a 1000+ dashboards (views) in the Search app. The test will be to see if that improves the performance.

0 Karma

vijaynukala
New Member

Any luck with that parameter?

0 Karma

seegeekrun
Path Finder

Yes. In fact, it was immediately better. Once it went out with the deploy, the initial response, TTFB, dropped from 10+ seconds to between 1-4.

In our UI cluster, we added the following to web.conf.
[settings]
...
max_view_cache_size = 2000
...

0 Karma

seegeekrun
Path Finder

I'm seeing similarly poor performance as well in the UI. The TTFB is very slow in the UI cluster. ~14s until First Byte.

We have theory that it may be due to the sheer number of dashboards in the Search app versus say specific applications that have been setup in the UI. But it's been challenging to nail down what the splunkweb process is looking to load in the initial request before it send it's first byte.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!