Deployment Architecture

Why is Splunk Web GUI Slow (search performance is fine, however)?

justinbarta
Explorer

The GUI is sooooo slow. If you run a search results are pulled in fast, the page is already loaded. The problem is when navigating around the GUI loading new pages, dashboards or even configuration settings. There is a significant delay in the server sending the html page back to the user.

Our old search head which is still online 6.3.2 is blazin fast. I'm not talking about searches I'm talking about page loads. Purely the hosting of the GUI.

We built a new search head cluster on new systems which are physical, SSD loaded, tons of RAM awesomeness.
Currently these systems are 6.5.0.
Even our stand alone non-clustered 6.5.0 search heads are slow.

Does anyone have an idea to this issue?
Does anyone have an idea where the web-hosting component of splunk is and how to adjust performance settings.
Else is there a way to decouple the hosting of the splunk GUI and move it to Apache etc?

Labels (2)

woodcock
Esteemed Legend

My guess would be that you were using init.d on the old server but (maybe without even knowing/deciding) switched to systemd on the new server.  The new systemd unit file is a disaster and I had a client lose half a year chasing performance problems because of EXACTLY this.  I would switch back to init.d and be happy.  See here:
https://community.splunk.com/t5/Monitoring-Splunk/Version-7-2-6-how-to-verify-systemctl-restart-stop...

0 Karma

koshyk
Super Champion

hi,
Any chance to get bit more information
- which Browser you use?
- If you have Chrome -> Developer tools -> Network. Please check if any javascripts are blocked by your firewalls/proxies
- Can you please provide an output of Job Inspector (dispatch.fetch , dispatch.stream.remote, command.search.rawdata)

0 Karma

seegeekrun
Path Finder

Version 64.0.3282.186 (Official Build) (64-bit)

Nothing is blocked. The waterfall is clean, so to speak, everything loads.

No Job Inspector since this is the initial load of the Search app, i.e. no search has even run. Which is why, I can see that the issue is sitting on the server (search head) side. The time to first byte is how long the browser is waiting for Splunk (splunkweb process) to process the and sent back the response.

So the question is, from a web framework point of view, what does Splunk do on the initial load. It would seem that there's zero caching of UI elements and it's generating everything each time.

0 Karma

seegeekrun
Path Finder

As an update, I'm going to try max_view_cache_size in web.conf. We have a 1000+ dashboards (views) in the Search app. The test will be to see if that improves the performance.

0 Karma

vijaynukala
New Member

Any luck with that parameter?

0 Karma

seegeekrun
Path Finder

Yes. In fact, it was immediately better. Once it went out with the deploy, the initial response, TTFB, dropped from 10+ seconds to between 1-4.

In our UI cluster, we added the following to web.conf.
[settings]
...
max_view_cache_size = 2000
...

0 Karma

Sudarshan77
New Member

Hi @seegeekrun  , i have updated the same to max_view_cache_size = 2000 under  path /apps/phantom/splunk/etc/system/default/web.conf

after this number change would need any service restart or reboot ?

0 Karma

seegeekrun
Path Finder

I'm seeing similarly poor performance as well in the UI. The TTFB is very slow in the UI cluster. ~14s until First Byte.

We have theory that it may be due to the sheer number of dashboards in the Search app versus say specific applications that have been setup in the UI. But it's been challenging to nail down what the splunkweb process is looking to load in the initial request before it send it's first byte.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...