topic Re: Splunk Dashboard Causing Browsers to crash? in Security

Splunk Dashboard Causing Browsers to crash?

butzowj — Mon, 04 Apr 2016 12:27:15 GMT

Hello,

We have Splunk 6.2, and we have a dashboard that utilizes 9 real-time searches and 4 historical searches. On both Chrome and Firefox, the dashboard causes the browser to crash intermittently throughout the day.

Is there anything that can be done to alleviate this issue? We'd like to keep the dashboard running throughout the day in our office, and this issue is preventing us from doing so.

Thanks,
JB

Re: Splunk Dashboard Causing Browsers to crash?

woodcock — Mon, 04 Apr 2016 12:55:40 GMT

Yes:

1: upgrade to the latest version (there are big improvements in parallelization).
2: stop using real-time searches (they probably are not working the way you expect anyway).

Re: Splunk Dashboard Causing Browsers to crash?

butzowj — Mon, 04 Apr 2016 13:29:17 GMT

Can you explain what is meant by "they probably are not working the way you expect anyway"?

Re: Splunk Dashboard Causing Browsers to crash?

woodcock — Mon, 04 Apr 2016 14:09:39 GMT

It is too varied but I would be happy to come in on a 1 day contract to explore it with you. It may suffice to say that in all of my work with clients, I have never seen a real-time search implementation that was working as desired and in the end, as we explored the requirements, we always used something non-real-time that was more than good enough, so that we could avoid the calamitous downsides of real-time.

Re: Splunk Dashboard Causing Browsers to crash?

woodcock — Mon, 04 Apr 2016 14:12:18 GMT

The only way to "make real-time work" is to mis-timestamp your events, which obviously, most people are unwilling to do.

Re: Splunk Dashboard Causing Browsers to crash?

halr9000 — Mon, 04 Apr 2016 14:55:02 GMT

I feel we would be better served to call this out as failures in design, not bad user behavior. Ideally, Splunk should NEVER cause the client to crash.

Re: Splunk Dashboard Causing Browsers to crash?

alacercogitatus — Mon, 04 Apr 2016 14:59:32 GMT

A few things:

1) You can force an automated refresh of the dashboard to help clear memory issues.
2) Re-design your Searches. Splunk searches consume memory in the browser. While not necessarily related to real-time, make sure that your searches are highly optimized. Ditch the real-time (as @woodcock mentions) and go with a historical search that only looks at the relevant time frame (last hour or so if that works for you).

To force automatic dashboard refreshes, edit the dashboard XML, and add this:

<dashboard refresh="300">

This will auto refresh the dashboard every 5 minutes (300 seconds).

There is also more information here: http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches on how to start optimizing your searches for efficiency.

Find us in IRC on efnet.org in #splunk, or join Slack (www.splunk402.com/chat) and find us there as well for all of your questions!

Re: Splunk Dashboard Causing Browsers to crash?

butzowj — Mon, 04 Apr 2016 15:02:17 GMT

This works for now - it may not be the best practices solution in the long term, but forcing the dashboard to refresh will definitely alleviate the crashes.

Thanks!

Re: Splunk Dashboard Causing Browsers to crash?

woodcock — Mon, 04 Apr 2016 15:05:47 GMT

When there is no more RAM, if ANY process running on the machine is not 100% bug free as regards memory-management, you will crash everything. So the memory-management problem could be ANYWHERE (in any running process) but the precipitating factor causing all RAM to be used is the real-time search load.

IMHO, if somebody has a legitimate real-time requirement, then Splunk is the wrong tool for the job.

This is because Splunk is "too literal" (this is a VERY good thing) as regards _time and because there is always latency everywhere in everything. The only way to claim to be "real-time" is to gloss over all latencies and pretend that things happened later than they really did. If Splunk had a pseudo-real-time mode that worked on _indextime then that would be something else entirely. I do not like to tell my data to lie to me; and I don't help my clients to tell their data to lie to them, either.

Re: Splunk Dashboard Causing Browsers to crash?

piebob — Mon, 04 Apr 2016 17:56:15 GMT

I downvoted this post because you are using answers to advertise your services. please don't do this in the future.

Re: Splunk Dashboard Causing Browsers to crash?

woodcock — Mon, 04 Apr 2016 18:18:01 GMT

Would it have been "OK" to say something like this:

You should see if you can hire a Splunk consultant for a couple of days to health-check your real-time searches because it is too complicated to go through in this forum

I was NOT trying to sell my services (I am plenty busy) but I was trying to give him a viable option because I think his dashboard is probably a ticking time-bomb on multiple fronts and it is not practical to unwrap it here.

Re: Splunk Dashboard Causing Browsers to crash?

piebob — Mon, 04 Apr 2016 18:46:15 GMT

that wording is definitely better. i appreciate your willingness to amend it. thanks.

Re: Splunk Dashboard Causing Browsers to crash?

sylim_splunk — Mon, 09 Jan 2017 04:16:44 GMT

This has been a known issue but recently splunkjs memory leak issue's been resolved and the fix is included in 6.4.5+ and 6.5.2+. Please try and provide any feedback.

Re: Splunk Dashboard Causing Browsers to crash?

johnpof — Fri, 03 Feb 2017 00:04:35 GMT

The memory leak is still there, I just upgraded my entire environment (10 nodes) to 6.5.2 mainly because I wanted this fixed and they are still running out of memory.