Memory leak on some dashboards

[mscomms]

Hi all, I am running splunk enterprise 8.1.2 and have a real headscratcher of a memory leak issue on dashboards running DataTables

1st Dashboard has a memory leak that causes the 10g memory of a pc to be used up in about 3 days
2nd Dashboard appears to have no leakage.

If I swap the xml between the dashboards the leak doesn't move with the xml but stays with the 1st dashboard and dashboard 2 stays leak free.

The xml for both dashboards except for very minor differences is identical and both dashboards use the same js scripts to generate the DataTables.

If I create a brand new dashboard and copy in the "leaky" xml and save it the dashboard leaks memory but if I create another brand new dashboard and copy in the "good" xml originally from dashboard 2 there is no memory leak 

"Good xml"

<form script="index.js" stylesheet="style.css" hideFilters="true">
  <label>Alert Overview</label>
  <search id="dataTableSearch">
    <query>| inputlookup noc_alerts_kv |search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$  | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$  | eval Key = _key  | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object |$sort$</query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <refresh>$refreshrate$</refresh>
    <sampleRatio>1</sampleRatio>
  </search>
  <fieldset autoRun="true" submitButton="true">
    <input type="dropdown" token="font">
      <label>Fontsize</label>
      <choice value="x-small">x-small</choice>
      <choice value="small">small</choice>
      <choice value="medium">medium</choice>
      <choice value="large">large</choice>
      <choice value="x-large">x-large</choice>
      <choice value="xx-large">xx-large</choice>
      <choice value="xxx-large">xxx-large</choice>
      <default>medium</default>
      <initialValue>medium</initialValue>
    </input>
    <input type="multiselect" token="fields">
      <label>Show Columns</label>
      <default>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,Severity,State,Key,SnowID</default>
      <initialValue>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,State,Severity,Key,SnowID</initialValue>
      <fieldForLabel>Fields</fieldForLabel>
      <fieldForValue>Fields</fieldForValue>
      <search>
        <query>| inputlookup AlertFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter> </delimiter>
    </input>
    <input type="dropdown" token="sort" searchWhenChanged="true">
      <label>Default Sort</label>
      <default>sort - DateTime</default>
      <initialValue>sort - DateTime</initialValue>
      <fieldForLabel>SortField</fieldForLabel>
      <fieldForValue>SortQuery</fieldForValue>
      <search>
        <query>| inputlookup SortFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="dropdown" token="refreshrate">
      <label>Refresh Rate</label>
      <choice value="10">10 secs</choice>
      <choice value="15">15 secs</choice>
      <choice value="20">20 secs</choice>
      <choice value="30">30 secs</choice>
      <choice value="45">45 secs</choice>
      <choice value="60">1 min</choice>
      <choice value="120">2 mins</choice>
      <choice value="300">5 mins</choice>
      <choice value="3600">Hourly</choice>
      <choice value="86400">Daily</choice>
      <default>60</default>
      <initialValue>60</initialValue>
    </input>
    <input type="multiselect" token="severity">
      <label>Severity</label>
      <valuePrefix>Severity =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DropdownSev</fieldForLabel>
      <fieldForValue>DropdownSev</fieldForValue>
      <search>
        <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="multiselect" token="datasource">
      <label>Sources</label>
      <valuePrefix>Datasource =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DTSource</fieldForLabel>
      <fieldForValue>DTSource</fieldForValue>
      <search>
        <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="checkbox" token="OpenState">
      <label></label>
      <choice value="*">Include Acknowledged</choice>
      <change>
        <condition label="Include Acknowledged">
          <set token="OpenState">OR State=Acknowledged</set>
        </condition>
        <condition>
          <set token="OpenState"></set>
        </condition>
      </change>
    </input>
    <input type="text" token="textsearch" searchWhenChanged="false">
      <label>Freeform Text</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <html>
      <img src="/static/app/noc/images/xxx.jpg"  style="width:359px; height:85px;"/>
   
  </html>
    </panel>
  </row>
  <row>
    <panel>
      <html>
        

      <div id="selectedrowcount"/>
      <button id="selectall" class="btn btn-primary">Select All</button>
      <button id="unselectall" class="btn btn-primary">Unselect All</button>
      <button id="acknow" class="btn btn-primary">Acknowledge</button>
      <button id="close" class="btn btn-primary">Close</button>
      <button id="submitInc" class="btn btn-primary">Add SNOW No.</button>
      <button id="raiseInc" class="btn btn-primary">Raise SNOW</button>
      <button id="getEventQuery" class="btn btn-primary">Get Event Data</button>
      <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button>
      <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button>
      <button id="pingIP" class="btn btn-primary">Ping IP</button>
      <div id="tableWrapper"/>
    </html>
    </panel>
  </row>
  <row depends="$htmlevent$">
    <panel>
      <html id="htmlevent">
      <label>eventdate Stuff do not delete</label>
        <div id="eventdata"/>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="snowfields">
        <div id="snowcont">
          <br/>
          <input type="radio" id="optinc" name="snowtype" value="Incident"/>
          <label for="incident">Incident</label>
          <input type="radio" id="optcase" name="snowtype" value="Case"/>
          <label for="case">Case</label>
          <label for="fsev">Severity:</label>
          <input type="text" id="sevfield" name="fsev"/>
          <label for="fshortdesc">Short Description:</label>
          <input type="text" id="shortdesc" name="fshortdesc"/>
          <label for="fdescription">Description:</label>
          <input type="text" id="descriptiontext" name="fdescription"/>
          <label for="fgroup">Group:</label>
          <select name="fgroup" id="assigngroup">
            <option value="noc">Telecoms Network Operations Centre</option>
            <option value="support">Telecoms Network Operations Centre Support Admin</option>
          </select>
          <label for="fcat">Category:</label>
          <select name="fcat" id="cat">
            <option value="none"/>
            <option value="datanetwork">Data Network</option>
            <option value="transmission">Transmission Network</option>
            <option value="infrastructure">Infrastructure</option>
            <option value="networkmanagement">Network Management</option>
            <option value="thirdparty">Third Party</option>
            <option value="itsystems">IT Systems</option>
            <option value="circuit">Circuit</option>
            <option value="security">Security</option>
            <option value="sites">Sites</option>
            <option value="controlsystems">Control Systems</option>
            <option value="mobile">Mobile</option>
            <option value="video">Video</option>
            <option value="voicerec">Voice Recording</option>
            <option value="voice">Voice</option>
            <option value="cti">CTI</option>
            <option value="reporting">Reporting</option>
          </select>
          <label for="fsubcat">Sub Category:</label>
          <select name="fsubcat" id="subcat">
           <option value="none"/>
          </select>
          <label for="fcasetype">Case Fault Type:</label>
          <select name="fcasetype" id="casefaulttype">
            <option value="none">--None--</option>
            <option value="servicedown">Service Down</option>
            <option value="degradedserv">Degraded Service</option>
            <option value="rfo">RFO</option>
            <option value="doa">DOA</option>
            <option value="telephony">Telephony</option>
          </select>
          <label for="fsla">SLA:</label>
          <select name="fsla" id="sla">
            
          </select>
          <label for="fcmdb">CMDB CI:</label>
          <input type="text" id="cmdbci" name="fcmdb"/>
          <br/>
          <button id="submitsnow" class="btn btn-primary">Submit</button>
          <button id="cancelsnow" class="btn btn-primary">Cancel</button>
        </div>
      </html>
    </panel>
  </row>
</form>

"Leaky xml"

<form script="index.js" stylesheet="style.css" hideFilters="true">
  <label>NOC Radio New Alarms</label>
  <search id="dataTableSearch">
    <query>| inputlookup noc_alerts_kv  | search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$  | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$  | eval Key = _key  | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object | $sort$</query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <refresh>$refreshrate$</refresh>
    <sampleRatio>1</sampleRatio>
  </search>
  <fieldset autoRun="true" submitButton="true">
    <input type="dropdown" token="font">
      <label>Fontsize</label>
      <choice value="x-small">x-small</choice>
      <choice value="small">small</choice>
      <choice value="medium">medium</choice>
      <choice value="large">large</choice>
      <choice value="x-large">x-large</choice>
      <choice value="xx-large">xx-large</choice>
      <choice value="xxx-large">xxx-large</choice>
      <default>medium</default>
      <initialValue>medium</initialValue>
    </input>
    <input type="multiselect" token="fields">
      <label>Show Columns</label>
      <default>AlertText,Datasource,DateTime,SnowID</default>
      <initialValue>AlertText,Datasource,DateTime,SnowID</initialValue>
      <fieldForLabel>Fields</fieldForLabel>
      <fieldForValue>Fields</fieldForValue>
      <search>
        <query>| inputlookup AlertFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter> </delimiter>
    </input>
    <input type="dropdown" token="sort" searchWhenChanged="true">
      <label>Default Sort</label>
      <default>sort - DateTime</default>
      <initialValue>sort - DateTime</initialValue>
      <fieldForLabel>SortField</fieldForLabel>
      <fieldForValue>SortQuery</fieldForValue>
      <search>
        <query>| inputlookup SortFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="dropdown" token="refreshrate">
      <label>Refresh Rate</label>
      <choice value="10">10 secs</choice>
      <choice value="15">15 secs</choice>
      <choice value="20">20 secs</choice>
      <choice value="30">30 secs</choice>
      <choice value="45">45 secs</choice>
      <choice value="60">1 min</choice>
      <choice value="120">2 mins</choice>
      <choice value="300">5 mins</choice>
      <choice value="3600">Hourly</choice>
      <choice value="86400">Daily</choice>
      <default>30</default>
      <initialValue>30</initialValue>
    </input>
    <input type="multiselect" token="severity">
      <label>Severity</label>
      <valuePrefix>Severity =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DropdownSev</fieldForLabel>
      <fieldForValue>DropdownSev</fieldForValue>
      <search>
        <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="multiselect" token="datasource">
      <label>Sources</label>
      <valuePrefix>Datasource =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DTSource</fieldForLabel>
      <fieldForValue>DTSource</fieldForValue>
      <search>
        <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>MDSRadio,Siaeradio</default>
      <initialValue>Siaeradio,MDSRadio</initialValue>
    </input>
    <input type="checkbox" token="OpenState">
      <label></label>
      <choice value="*">Include Acknowledged</choice>
      <change>
        <condition label="Include Acknowledged">
          <set token="OpenState">OR State=Acknowledged</set>
        </condition>
        <condition>
          <set token="OpenState"></set>
        </condition>
      </change>
      <delimiter> </delimiter>
    </input>
    <input type="text" token="textsearch" searchWhenChanged="false">
      <label>Freeform Text</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <html>
      <img src="/static/app/noc/images/xxx.jpg"  style="width:359px; height:85px;"/>
  </html>
    </panel>
  </row>
  <row>
    <panel>
      <html>
        
      <div id="tableWrapper"/>
      <div id="selectedrowcount"/>
      <button id="selectall" class="btn btn-primary">Select All</button>
      <button id="unselectall" class="btn btn-primary">Unselect All</button>
      <button id="acknow" class="btn btn-primary">Acknowledge</button>
      <button id="close" class="btn btn-primary">Close</button>
      <button id="submitInc" class="btn btn-primary">Add SNOW No.</button>
      <button id="raiseInc" class="btn btn-primary">Raise SNOW</button>
      <button id="getEventQuery" class="btn btn-primary">Get Event Data</button>
      <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button>
      <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button>
      <button id="pingIP" class="btn btn-primary">Ping IP</button>

    </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="htmlevent">
        <div id="eventdata"/>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="snowfields">
        <div id="snowcont"><br />
          <input type="radio" id="optinc" name="snowtype" value="Incident" />
          <label for="incident">Incident</label>
          <input type="radio" id="optcase" name="snowtype" value="Case"  />
          <label for="case">Case</label>
          <label for="fsev">Severity:</label>
          <input type="text" id="sevfield" name="fsev"  />
          <label for="fshortdesc">Short Description:</label>
          <input type="text" id="shortdesc" name="fshortdesc"  />
          <label for="fdescription">Description:</label>
          <input type="text" id="descriptiontext" name="fdescription" />
          <label for="fgroup">Group:</label>
          <select name="fgroup" id="assigngroup">
            <option value="noc">Telecoms Network Operations Centre</option>
            <option value="support">Telecoms Network Operations Centre Support Admin</option>
          </select>
          <label for="fcat">Category:</label>
          <select name="fcat" id="cat">
            <option value="none"></option>
            <option value="datanetwork">Data Network</option>
            <option value="transmission">Transmission Network</option>
            <option value="infrastructure">Infrastructure</option>
            <option value="networkmanagement">Network Management</option>
            <option value="thirdparty">Third Party</option>
            <option value="itsystems">IT Systems</option>
            <option value="circuit">Circuit</option>
            <option value="security">Security</option>
            <option value="sites">Sites</option>
            <option value="controlsystems">Control Systems</option>
            <option value="mobile">Mobile</option>
            <option value="video">Video</option>
            <option value="voicerec">Voice Recording</option>
            <option value="voice">Voice</option>
            <option value="cti">CTI</option>
            <option value="reporting">Reporting</option>
          </select>
          <label for="fsubcat">Sub Category:</label>
          <select name="fsubcat" id="subcat">
           <option value="none"></option>
          </select>
          <label for="fcasetype">Case Fault Type:</label>
          <select name="fcasetype" id="casefaulttype">
            <option value="none">--None--</option>
            <option value="servicedown">Service Down</option>
            <option value="degradedserv">Degraded Service</option>
            <option value="rfo">RFO</option>
            <option value="doa">DOA</option>
            <option value="telephony">Telephony</option>
          </select>
          <label for="fsla">SLA:</label>
          <select name="fsla" id="sla">
            
          </select>
          <label for="fcmdb">CMDB CI:</label>
          <input type="text" id="cmdbci" name="fcmdb"  /><br />
          <button id="submitsnow" class="btn btn-primary">Submit</button>
          <button id="cancelsnow" class="btn btn-primary">Cancel</button>
        </div>
      </html>
    </panel>
  </row>
</form>

Anyone got any ideas?

[mscomms]

The issue appears to occur when you hide columns using the .column(i).visible(false) method as soon as you hide a column the DataTable starts leaking memory I have worked around this using CSS to hide the columns

[mscomms]

Update

If you create a new empty dashboard and make a trivial change to it then save it, then edit it and paste in the leaky xml the dashboard doesn't leak

[ivan_mirosav]

Hi mscomms,

I'm experiencing much the same behavior from a memory standpoint. Can you show me the search you used to identify the leaky dashboard?

[ivan_mirosav]

Very useful! I see now that you mentioned this is eating PC memory. My issue is similar in behavior but in regards to Search Head memory. I'll keep this tool in mind for browser memory consumption on user PCs.

Thanks!

[mscomms]

Hi Ivan, 

I opened the dashboard as normal then hit F12 to open DevTools, go to the Memory tab and hit start in the bottom left of the screen you will see a page like this

let it run and you will see peaks of memory useage when the dashboard runs its query. Immediatly after running the peaks will be filled with a colour (in my case blue) which should clear as the dashboard reruns its query. Ig the spikes stay full yopu are leaking memory

Thisd is my non leaky dashboard

and this is the leaky one

once it has captured some data click the red button top left and let the capture process, once it has completed processing you can inspect what is using the memory which will hopefully give you and idea where to investigate.