Splunk Web performance slow for VM

[mcrawford44]

All,

I've noticed that web performance has been slow since installing Splunk, however I brushed it off as a hardware issue since we were on a small VM for pilot.

Once migrated to a 16vcpu server with 128gb of ram I expected performance to increase. It has not.

I've spent a couple days rummaging through configuration files and increasing CherryPy thread settings and the allocated ram for DBX, but the interface is still very slow.

Is there an optimization guide I'm missing or an obvious fix I'm missing?

[sideview]

At a high level, reporting and UI slowness is more often caused by less-than-ideal disk IO, rather than CPU and ram and this goes double for VM's. Splunk reads a lot from disk and has to do it fast for reports to be fast. VM's suffer just because all else being equal they usually have less IO throughput than a physical host. Check out this what this page has to say under "Disk subsystem" http://docs.splunk.com/Documentation/Splunk/6.1.1/Installation/Referencehardware

[mcrawford44]

I have just started bench-marking the disks and I believe that is the cause. Our VM OS drive is extremely low powered for some reason.

[martin_mueller]

To isolate the issue you could run a browser on the server locally and connect to itself. If that's slow then it's something on the server, if that's fast then it's something in the network.

[mcrawford44]

I don't believe so. Tried multiple browsers across several variations of machines, from i3 laptops to i7 beasts.

I installed Splunk on my local machine and it runs as fast as expected. I'm having our network guys take a look at it.

[martin_mueller]

Is it maybe your browser that's slow? You seem certain to rummage around the server, but I'd check that first.