Why is Splunk Web GUI Slow (search performance is fine, however)?

[justinbarta]

The GUI is sooooo slow. If you run a search results are pulled in fast, the page is already loaded. The problem is when navigating around the GUI loading new pages, dashboards or even configuration settings. There is a significant delay in the server sending the html page back to the user.

Our old search head which is still online 6.3.2 is blazin fast. I'm not talking about searches I'm talking about page loads. Purely the hosting of the GUI.

We built a new search head cluster on new systems which are physical, SSD loaded, tons of RAM awesomeness.
Currently these systems are 6.5.0.
Even our stand alone non-clustered 6.5.0 search heads are slow.

Does anyone have an idea to this issue?
Does anyone have an idea where the web-hosting component of splunk is and how to adjust performance settings.
Else is there a way to decouple the hosting of the splunk GUI and move it to Apache etc?

[woodcock]

My guess would be that you were using init.d on the old server but (maybe without even knowing/deciding) switched to systemd on the new server.  The new systemd unit file is a disaster and I had a client lose half a year chasing performance problems because of EXACTLY this.  I would switch back to init.d and be happy.  See here:
https://community.splunk.com/t5/Monitoring-Splunk/Version-7-2-6-how-to-verify-systemctl-restart-stop...

[koshyk]

hi,
Any chance to get bit more information
- which Browser you use?
- If you have Chrome -> Developer tools -> Network. Please check if any javascripts are blocked by your firewalls/proxies
- Can you please provide an output of Job Inspector (dispatch.fetch , dispatch.stream.remote, command.search.rawdata)

[seegeekrun]

Version 64.0.3282.186 (Official Build) (64-bit)

Nothing is blocked. The waterfall is clean, so to speak, everything loads.

No Job Inspector since this is the initial load of the Search app, i.e. no search has even run. Which is why, I can see that the issue is sitting on the server (search head) side. The time to first byte is how long the browser is waiting for Splunk (splunkweb process) to process the and sent back the response.

So the question is, from a web framework point of view, what does Splunk do on the initial load. It would seem that there's zero caching of UI elements and it's generating everything each time.

[seegeekrun]

As an update, I'm going to try max_view_cache_size in web.conf. We have a 1000+ dashboards (views) in the Search app. The test will be to see if that improves the performance.

[vijaynukala]

Any luck with that parameter?

[seegeekrun]

Yes. In fact, it was immediately better. Once it went out with the deploy, the initial response, TTFB, dropped from 10+ seconds to between 1-4.

In our UI cluster, we added the following to web.conf.
[settings]
...
max_view_cache_size = 2000
...

[Sudarshan77]

Hi @seegeekrun  , i have updated the same to max_view_cache_size = 2000 under  path /apps/phantom/splunk/etc/system/default/web.conf

after this number change would need any service restart or reboot ?

[seegeekrun]

I'm seeing similarly poor performance as well in the UI. The TTFB is very slow in the UI cluster. ~14s until First Byte.

We have theory that it may be due to the sheer number of dashboards in the Search app versus say specific applications that have been setup in the UI. But it's been challenging to nail down what the splunkweb process is looking to load in the initial request before it send it's first byte.