Security

Splunk Dashboard Causing Browsers to crash?

butzowj
Path Finder

Hello,

We have Splunk 6.2, and we have a dashboard that utilizes 9 real-time searches and 4 historical searches. On both Chrome and Firefox, the dashboard causes the browser to crash intermittently throughout the day.

Is there anything that can be done to alleviate this issue? We'd like to keep the dashboard running throughout the day in our office, and this issue is preventing us from doing so.

Thanks,
JB

1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

A few things:

1) You can force an automated refresh of the dashboard to help clear memory issues.
2) Re-design your Searches. Splunk searches consume memory in the browser. While not necessarily related to real-time, make sure that your searches are highly optimized. Ditch the real-time (as @woodcock mentions) and go with a historical search that only looks at the relevant time frame (last hour or so if that works for you).

To force automatic dashboard refreshes, edit the dashboard XML, and add this:

<dashboard refresh="300">

This will auto refresh the dashboard every 5 minutes (300 seconds).

There is also more information here: http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches on how to start optimizing your searches for efficiency.

Find us in IRC on efnet.org in #splunk, or join Slack (www.splunk402.com/chat) and find us there as well for all of your questions!

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

This has been a known issue but recently splunkjs memory leak issue's been resolved and the fix is included in 6.4.5+ and 6.5.2+. Please try and provide any feedback.

0 Karma

johnpof
Path Finder

The memory leak is still there, I just upgraded my entire environment (10 nodes) to 6.5.2 mainly because I wanted this fixed and they are still running out of memory.

alacercogitatus
SplunkTrust
SplunkTrust

A few things:

1) You can force an automated refresh of the dashboard to help clear memory issues.
2) Re-design your Searches. Splunk searches consume memory in the browser. While not necessarily related to real-time, make sure that your searches are highly optimized. Ditch the real-time (as @woodcock mentions) and go with a historical search that only looks at the relevant time frame (last hour or so if that works for you).

To force automatic dashboard refreshes, edit the dashboard XML, and add this:

<dashboard refresh="300">

This will auto refresh the dashboard every 5 minutes (300 seconds).

There is also more information here: http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches on how to start optimizing your searches for efficiency.

Find us in IRC on efnet.org in #splunk, or join Slack (www.splunk402.com/chat) and find us there as well for all of your questions!

butzowj
Path Finder

This works for now - it may not be the best practices solution in the long term, but forcing the dashboard to refresh will definitely alleviate the crashes.

Thanks!

0 Karma

woodcock
Esteemed Legend

Yes:

1: upgrade to the latest version (there are big improvements in parallelization).
2: stop using real-time searches (they probably are not working the way you expect anyway).

0 Karma

halr9000
Motivator

I feel we would be better served to call this out as failures in design, not bad user behavior. Ideally, Splunk should NEVER cause the client to crash.

woodcock
Esteemed Legend

When there is no more RAM, if ANY process running on the machine is not 100% bug free as regards memory-management, you will crash everything. So the memory-management problem could be ANYWHERE (in any running process) but the precipitating factor causing all RAM to be used is the real-time search load.

IMHO, if somebody has a legitimate real-time requirement, then Splunk is the wrong tool for the job.

This is because Splunk is "too literal" (this is a VERY good thing) as regards _time and because there is always latency everywhere in everything. The only way to claim to be "real-time" is to gloss over all latencies and pretend that things happened later than they really did. If Splunk had a pseudo-real-time mode that worked on _indextime then that would be something else entirely. I do not like to tell my data to lie to me; and I don't help my clients to tell their data to lie to them, either.

0 Karma

butzowj
Path Finder

Can you explain what is meant by "they probably are not working the way you expect anyway"?

0 Karma

woodcock
Esteemed Legend

It is too varied but I would be happy to come in on a 1 day contract to explore it with you. It may suffice to say that in all of my work with clients, I have never seen a real-time search implementation that was working as desired and in the end, as we explored the requirements, we always used something non-real-time that was more than good enough, so that we could avoid the calamitous downsides of real-time.

0 Karma

piebob
Splunk Employee
Splunk Employee

I downvoted this post because you are using answers to advertise your services. please don't do this in the future.

woodcock
Esteemed Legend

Would it have been "OK" to say something like this:

You should see if you can hire a Splunk consultant for a couple of days to health-check your real-time searches because it is too complicated to go through in this forum

I was NOT trying to sell my services (I am plenty busy) but I was trying to give him a viable option because I think his dashboard is probably a ticking time-bomb on multiple fronts and it is not practical to unwrap it here.

0 Karma

piebob
Splunk Employee
Splunk Employee

that wording is definitely better. i appreciate your willingness to amend it. thanks.

0 Karma

woodcock
Esteemed Legend

The only way to "make real-time work" is to mis-timestamp your events, which obviously, most people are unwilling to do.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...