Splunk Dashboard Causing Browsers to crash?

[butzowj]

Hello,

We have Splunk 6.2, and we have a dashboard that utilizes 9 real-time searches and 4 historical searches. On both Chrome and Firefox, the dashboard causes the browser to crash intermittently throughout the day.

Is there anything that can be done to alleviate this issue? We'd like to keep the dashboard running throughout the day in our office, and this issue is preventing us from doing so.

Thanks,
JB

[sylim_splunk]

This has been a known issue but recently splunkjs memory leak issue's been resolved and the fix is included in 6.4.5+ and 6.5.2+. Please try and provide any feedback.

[johnpof]

The memory leak is still there, I just upgraded my entire environment (10 nodes) to 6.5.2 mainly because I wanted this fixed and they are still running out of memory.

[alacercogitatus · Accepted solution]

A few things:

1) You can force an automated refresh of the dashboard to help clear memory issues.
2) Re-design your Searches. Splunk searches consume memory in the browser. While not necessarily related to real-time, make sure that your searches are highly optimized. Ditch the real-time (as @woodcock mentions) and go with a historical search that only looks at the relevant time frame (last hour or so if that works for you).

To force automatic dashboard refreshes, edit the dashboard XML, and add this:

<dashboard refresh="300">

This will auto refresh the dashboard every 5 minutes (300 seconds).

There is also more information here: http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches on how to start optimizing your searches for efficiency.

Find us in IRC on efnet.org in #splunk, or join Slack (www.splunk402.com/chat) and find us there as well for all of your questions!

[butzowj]

This works for now - it may not be the best practices solution in the long term, but forcing the dashboard to refresh will definitely alleviate the crashes.

Thanks!

[woodcock]

Yes:

1: upgrade to the latest version (there are big improvements in parallelization).
2: stop using real-time searches (they probably are not working the way you expect anyway).

[halr9000]

I feel we would be better served to call this out as failures in design, not bad user behavior. Ideally, Splunk should NEVER cause the client to crash.

[woodcock]

When there is no more RAM, if ANY process running on the machine is not 100% bug free as regards memory-management, you will crash everything. So the memory-management problem could be ANYWHERE (in any running process) but the precipitating factor causing all RAM to be used is the real-time search load.

IMHO, if somebody has a legitimate real-time requirement, then Splunk is the wrong tool for the job.

This is because Splunk is "too literal" (this is a VERY good thing) as regards _time and because there is always latency everywhere in everything. The only way to claim to be "real-time" is to gloss over all latencies and pretend that things happened later than they really did. If Splunk had a pseudo-real-time mode that worked on _indextime then that would be something else entirely. I do not like to tell my data to lie to me; and I don't help my clients to tell their data to lie to them, either.

[butzowj]

Can you explain what is meant by "they probably are not working the way you expect anyway"?

[woodcock]

It is too varied but I would be happy to come in on a 1 day contract to explore it with you. It may suffice to say that in all of my work with clients, I have never seen a real-time search implementation that was working as desired and in the end, as we explored the requirements, we always used something non-real-time that was more than good enough, so that we could avoid the calamitous downsides of real-time.

[piebob]

I downvoted this post because you are using answers to advertise your services. please don't do this in the future.

[woodcock]

Would it have been "OK" to say something like this:

You should see if you can hire a Splunk consultant for a couple of days to health-check your real-time searches because it is too complicated to go through in this forum

I was NOT trying to sell my services (I am plenty busy) but I was trying to give him a viable option because I think his dashboard is probably a ticking time-bomb on multiple fronts and it is not practical to unwrap it here.

[piebob]

that wording is definitely better. i appreciate your willingness to amend it. thanks.

[woodcock]

The only way to "make real-time work" is to mis-timestamp your events, which obviously, most people are unwilling to do.