Dashboards & Visualizations

Why does Splunk dashboard Javascript cause Firefox to to increase memory usage until it hangs and needs to be restarted?

rberse
Explorer

When opening our Splunk dashboard in Firefox (v 50.0.2 on Windows 7), the amount of RAM used by Firefox gradually increases until it hits 4GB and then Firefox hangs and needs to be restarted. This happens consistently on multiple machines. I was able to use Firefox's about:memory tool to identify Splunk as the culprit.

I seem to be unable to upload a screenshot of the about:memory output, but at the time I took the snapshot, the dashboard window was using 1.5GB of memory, with the bulk of it under "js-compartment", and the majority of that until "class(Object)/objects". A little more than half of this was in the malloc-heap with the remainder in the gc-heap.

The 1.5GB will just keep increasing until FF needs to be restarted. We are not able to use Internet Explorer or Chrome, and so need a solution for Firefox. Our Splunk version is 6.5.0.

Please advise.

0 Karma

skalliger
SplunkTrust
SplunkTrust

Yep, we had a similar problem when we did an upgrade to 6.5. Fortunately, they changed the refresh function. Earlier, the complete panels were reloaded and that's why we used real-time searches sometimes. The real-time searches in 6.5 ate up all the memory, no matter how many RAM was installed. So, simply use the refresh now with 6.5 as @niketnilay said. 🙂

0 Karma

sheamus69
Communicator

Out of interest, what version of Firefox are you runnning? I've seen some odd memory issues in the past from older versions of firefox, so the question is worth asking...

0 Karma

rberse
Explorer

50, generally we are running the latest version or close to it.

0 Karma

niketn
Legend

@rberse the issue could be because of Real Time queries. Alternatively try to see if you can Run dasahboard queries without Realtime time span and refresh required panels of dashboard in near real-time kind of way like every one minute. (Refreshing individual Dashboard panel is a feature of Splunk 6.5.x).

     <earliest>-24h@h</earliest>
     <latest>now</latest>

Instead of the following:

 <earliest>rt-24h@h</earliest>
 <latest>rtnow</latest>

In case it is required to run Real Time dashboard (which would not be a recommended approach) you would need to upgrade your underlying Splunk Infrastructure hardware.

PS: You would also need to include all the search optimization techniques for improving query performance and reducing number of events pulled/loaded in the dashboard.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

rberse
Explorer

Thanks, @niketnilay, I will try changing this to non-realtime. According to the documentation, the refresh option for individual panel elements (refresh.auto.interval) is deprecated. What about simply refreshing the entire form/dashboard or refreshing the search in each panel?

We ideally need real-time elements in our dashboard. Why would the underlying Splunk hardware have any effect on the amount of memory Javascript is using on the browser side? Also, I can see the advantage of optimizing query performance, but why would this impact memory usage in the browser?

0 Karma

niketn
Legend

Yes refresh.auto.interval is deprecated. However, I was asking for the attributes which have been newly introduced in Splunk 6.5 version:

  <refresh>
  <refreshType>

http://docs.splunk.com/Documentation/Splunk/6.5.1/Viz/PanelreferenceforSimplifiedXML#search

Realtime searches can put a load on any browsers not just Firefox. It is CPU/Memory intrinsic operation. Refer to documentation below: https://docs.splunk.com/Documentation/Splunk/6.5.1/Search/Realtimeperformanceandlimitations

Moreover, I just wanted to be sure whether the hanging issue is because of Real Time Searches or not. You can try removing real time searches and see if you still face the issue with Panel/Dashboard refresh, whichever suits your needs.

If you find that your browser is actually freezing due to real-time searches and not something else, then you can think of taking the next steps of designing properly for real time searches, in case you ablsolutely need it.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

niketn
Legend

Can you first check whether Real-Time searches are the culprit for memory issue or not? Also check whether it helps if you disable Fire Fox Browser caching.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

rberse
Explorer

Hi @niketnilay, thanks for the clarification. I am testing the changes now (removed rt searches and put a 15 second refresh on the search element in each panel). So far it looks promising, but I should be able to say for sure by the end of the day.

0 Karma

rberse
Explorer

Hi @nicketnilay- apologies for the long delay, I was on vacation. I took some time testing the dashboard without realtime queries, and it improved the issue, but did not eliminate it. The memory footprint still increases without bound until the browser hangs, but it does so more slowly.

I played around with some of the memory settings in firefox that should control the limit of RAM used for javascript, but that did not make a difference. When you talk about disabling browser caching- what setting can I use to control that, specifically? All the documentation I have seen seems to control caching files, rather than caching data in memory.

Thanks,
Ryan

0 Karma

niketn
Legend

One of the simplest way would be to try and run the dashboard in incognito mode.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

rberse
Explorer

I thought that incognito mode would disable only caching to files, whereas your suggestion was to disable the cache in memory. At any rate, I am giving it a try.

I did find the browser.cache.memory.enable setting in the FF config, and disable it. But that did not help.

0 Karma

rberse
Explorer

@niketnilay- unfortunately, running in incognito mode does not seem to make a difference 😞

So far, the only thing that has helped (but not solved) the problem is removing the real-time queries. Even with that, within the course of the work day, the browser will reliably hang due to memory consumption 😞

Any other ideas?

0 Karma

rjthibod
Champion

The community would likely need more information about the app and dashboard you are looking at. Is this a packaged app on splunkbase or a custom in-house app? Is the app loading any custom JS modules/classes?

I have a fairly complicated app with custom JS, and it does not present this issue in a similar environment, so there is likely something specific to the dashboard you are looking at.

0 Karma

rberse
Explorer

Thanks for your response, @rjthibod. The dashboard is custom built by us, but we did not use any custom JS. We built the dashboard by using the Edit Dashboard UI to build our various panels and then modifying/adding to the XML.

Is there something more specific I should provide in terms of information?

0 Karma

rjthibod
Champion

Are you using any custom visualizations / add-on visualizations in the dashboard? Is the dashboard built inside of a custom app or the default search app? If the former, is the app loading any JS by default?

Other than those generic questions, the more you can share about the raw XML, the better.

0 Karma

rberse
Explorer

Hi @rjthibod, apologies for the delayed response, I was tied up on other work. I have included the XML below, with a few things removed or changed (most of the queries have the middles replaced with '...' to save space). Essentially, we have a bunch of single value panes, which are clickable and which will take the user to a similar (not identical) query upon click. In addition, some of the panes have a checkbox which the user can click, and which will display a smaller preview of the data under the single number.

<?xml version="1.0" encoding="UTF-8"?>
<form>
  <label>Dashboard Title</label>
  <fieldset autoRun="true">
    <input type="time" searchWhenChanged="true">
      <label>Time period</label>
      <default>
        <earliest>rt-24h@h</earliest>
        <latest>rtnow</latest>
      </default>
    </input>
    <input type="text" token="rp_query" depends="$nonexistentField$">
      <default>
        <![CDATA[tag=ROUTING | ... | table _time host facility Neighbor_IP Interface Description latest_state_to latest_reason count]]>
      </default>
    </input>
    <input type="text" token="vpn_query" depends="$nonexistentField$">
      <default>crypto tunnel | ... | table _time host Neighbor_IP Description Latest_state count</default>
    </input>
    <input type="text" token="rt_query" depends="$nonexistentField$">
      <default>tag=RoutingTracking  | ... | table _time,host,Tracking_ID,Tracked_Route,Description, Latest_state count</default>
    </input>
    <input type="text" token="ls_query" depends="$nonexistentField$">
      <default>tag=LinkStatus | ... | table _time host Interface latest_state Description</default>
    </input>
    <input type="text" token="slbvrrp_query" depends="$nonexistentField$">
      <default>tag=VRRP| ... | table _time host VirtualRouter_IP Latest_state count</default>
    </input>
    <input type="text" token="slbportstate_query" depends="$nonexistentField$">
      <default>tag=XLinkStatus | ... |table _time host Interface Latest_state count</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Panel 1</title>
      <single>
        <title>Routing Protocols</title>
        <search>
          <query>tag=ROUTING | ... | stats count</query>
        </search>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="linkView">
          <![CDATA[/app/search/search?earliest=$earliest$&latest=$latest$&q=$rp_query$]]>
        </option>
        <option name="underLabel">Down</option>
        <option name="drilldown">all</option>
      </single>
      <single>
        <title>VPN Tunnels</title>
        <search>
          <query>crypto tunnel | ... | stats count</query>
        </search>
        <option name="linkFields">result</option>
        <option name="linkSearch"></option>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="linkView">
          <![CDATA[/app/search/search?earliest=$earliest$&latest=$latest$&q=$vpn_query$]]>
        </option>
        <option name="underLabel">Down</option>
        <option name="drilldown">all</option>
      </single>
      <single>
        <title>Route Tracking</title>
        <search>
          <query>tag=RoutingTracking | ... | stats count</query>
        </search>
        <option name="linkFields">result</option>
        <option name="linkSearch"></option>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="linkView">
          <![CDATA[/app/search/search?earliest=$earliest$&latest=$latest$&q=$rt_query$]]>
        </option>
        <option name="underLabel">Down</option>
        <option name="drilldown">all</option>
      </single>
    </panel>
    <panel>
      <title>VRRP Status</title>
      <single>
        <title>SLB VRRP</title>
        <search>
          <query>tag=VRRP host=*slb-01*| ... | stats count</query>
        </search>
        <option name="linkFields">result</option>
        <option name="linkSearch"></option>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="linkView">
          <![CDATA[/app/search/search?earliest=$earliest$&latest=$latest$&q=$slbvrrp_query$]]>
        </option>
        <option name="underLabel">Down</option>
        <option name="drilldown">all</option>
      </single>
      <single>
        <title>SLB Port Status</title>
        <search>
          <query>tag=LinkStatus | ... | stats count</query>
        </search>
        <option name="linkFields">result</option>
        <option name="linkSearch"></option>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="linkView">
          <![CDATA[/app/search/search?earliest=$earliest$&latest=$latest$&q=$slbvrrp_query$]]>
        </option>
        <option name="underLabel">Down</option>
        <option name="drilldown">all</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Panel2</title>
      <input type="checkbox" token="displayDetails2" searchWhenChanged="true">
        <label></label>
        <choice value="true">Display Details</choice>
      </input>
      <single>
        <search>
          <query>tag=Firewall |... | stats count</query>
        </search>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="drilldown">all</option>
        <option name="numberPrecision">0</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="underLabel">VRRP Interfaces That Changed State</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
      </single>
      <table depends="$displayDetails2$">
        <search>
          <query>tag=Firewall2 | ... | stats latest(_time) AS _time latest(mb_state) AS latest_mb_state latest(state) AS latest_state count AS state_changes by host Interface vrid</query>
        </search>
      </table>
    </panel>
    <panel>
      <title>Panel3</title>
      <input type="checkbox" token="displayDetails1" searchWhenChanged="true">
        <label></label>
        <choice value="true">Display Details</choice>
      </input>
      <single>
        <search>
          <query>tag=LinkStatus | ... | stats count</query>
        </search>
        <option name="drilldown">none</option>
        <option name="colorBy">value</option>
        <option name="colorMode">none</option>
        <option name="numberPrecision">0</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
        <option name="underLabel">Device Interfaces Down</option>
        <option name="linkFields">result</option>
        <option name="linkView">/app/search/search?earliest=$earliest$&amp;latest=$latest$&amp;q=$ls_query$</option>
        <option name="rangeColors">["0x65a637","0xd93f3c"]</option>
        <option name="rangeValues">[0]</option>
      </single>
      <table depends="$displayDetails1$">
        <search>
          <query>tag=LinkStatus |... | search latest_state="Down"</query>
        </search>
      </table>
    </panel>
  </row>
</form>
0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

You have probably search results in this dashboard returning events or lots of data so splunk probably cache some of them in the web page (so you can browse quickly when cliking 1,2,...even if most of the time users don't use this)

What you can do is try to limit this kind of results , limit number of results or use multiple dashboards for this with adequate drilldown instead of a big one .
you can also try in limits.conf remote_timeline_fetchall = 0 (read the spec to understand what it does and if that's interesting for you)

0 Karma

rjthibod
Champion

@rberse, I would agree with @maraman_splunk.

One thing to consider doing is removing panels/charts one at a time to see if you can isolate a specific, problematic item. In general, I have never used so many real-time searches, and your use of returning results to tables makes me suspect the caching issue is likely.

0 Karma

rberse
Explorer

Thanks @rjthibod and @marman_splunk. The caching idea sounds plausible, although I am surprised Firefox wouldn't place some kind of limit on it. I check out the setting in limits.conf and also see if I can disable this on the Firefox side.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...