Dashboards & Visualizations

Memory leak on some dashboards

mscomms
Path Finder

Hi all, I am running splunk enterprise 8.1.2 and have a real headscratcher of a memory leak issue on dashboards running DataTables

1st Dashboard has a memory leak that causes the 10g memory of a pc to be used up in about 3 days
2nd Dashboard appears to have no leakage.

If I swap the xml between the dashboards the leak doesn't move with the xml but stays with the 1st dashboard and dashboard 2 stays leak free.

The xml for both dashboards except for very minor differences is identical and both dashboards use the same js scripts to generate the DataTables.

If I create a brand new dashboard and copy in the "leaky" xml and save it the dashboard leaks memory but if I create another brand new dashboard and copy in the "good" xml originally from dashboard 2 there is no memory leak 

"Good xml"

 

<form script="index.js" stylesheet="style.css" hideFilters="true">
  <label>Alert Overview</label>
  <search id="dataTableSearch">
    <query>| inputlookup noc_alerts_kv |search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$  | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$  | eval Key = _key  | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object |$sort$</query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <refresh>$refreshrate$</refresh>
    <sampleRatio>1</sampleRatio>
  </search>
  <fieldset autoRun="true" submitButton="true">
    <input type="dropdown" token="font">
      <label>Fontsize</label>
      <choice value="x-small">x-small</choice>
      <choice value="small">small</choice>
      <choice value="medium">medium</choice>
      <choice value="large">large</choice>
      <choice value="x-large">x-large</choice>
      <choice value="xx-large">xx-large</choice>
      <choice value="xxx-large">xxx-large</choice>
      <default>medium</default>
      <initialValue>medium</initialValue>
    </input>
    <input type="multiselect" token="fields">
      <label>Show Columns</label>
      <default>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,Severity,State,Key,SnowID</default>
      <initialValue>AlertText,CustomerID,Datasource,DateTime,Hostname,IP_Address,Object,State,Severity,Key,SnowID</initialValue>
      <fieldForLabel>Fields</fieldForLabel>
      <fieldForValue>Fields</fieldForValue>
      <search>
        <query>| inputlookup AlertFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter> </delimiter>
    </input>
    <input type="dropdown" token="sort" searchWhenChanged="true">
      <label>Default Sort</label>
      <default>sort - DateTime</default>
      <initialValue>sort - DateTime</initialValue>
      <fieldForLabel>SortField</fieldForLabel>
      <fieldForValue>SortQuery</fieldForValue>
      <search>
        <query>| inputlookup SortFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="dropdown" token="refreshrate">
      <label>Refresh Rate</label>
      <choice value="10">10 secs</choice>
      <choice value="15">15 secs</choice>
      <choice value="20">20 secs</choice>
      <choice value="30">30 secs</choice>
      <choice value="45">45 secs</choice>
      <choice value="60">1 min</choice>
      <choice value="120">2 mins</choice>
      <choice value="300">5 mins</choice>
      <choice value="3600">Hourly</choice>
      <choice value="86400">Daily</choice>
      <default>60</default>
      <initialValue>60</initialValue>
    </input>
    <input type="multiselect" token="severity">
      <label>Severity</label>
      <valuePrefix>Severity =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DropdownSev</fieldForLabel>
      <fieldForValue>DropdownSev</fieldForValue>
      <search>
        <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="multiselect" token="datasource">
      <label>Sources</label>
      <valuePrefix>Datasource =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DTSource</fieldForLabel>
      <fieldForValue>DTSource</fieldForValue>
      <search>
        <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="checkbox" token="OpenState">
      <label></label>
      <choice value="*">Include Acknowledged</choice>
      <change>
        <condition label="Include Acknowledged">
          <set token="OpenState">OR State=Acknowledged</set>
        </condition>
        <condition>
          <set token="OpenState"></set>
        </condition>
      </change>
    </input>
    <input type="text" token="textsearch" searchWhenChanged="false">
      <label>Freeform Text</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <html>
      <img src="/static/app/noc/images/xxx.jpg"  style="width:359px; height:85px;"/>
   
  </html>
    </panel>
  </row>
  <row>
    <panel>
      <html>
        

      <div id="selectedrowcount"/>
      <button id="selectall" class="btn btn-primary">Select All</button>
      <button id="unselectall" class="btn btn-primary">Unselect All</button>
      <button id="acknow" class="btn btn-primary">Acknowledge</button>
      <button id="close" class="btn btn-primary">Close</button>
      <button id="submitInc" class="btn btn-primary">Add SNOW No.</button>
      <button id="raiseInc" class="btn btn-primary">Raise SNOW</button>
      <button id="getEventQuery" class="btn btn-primary">Get Event Data</button>
      <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button>
      <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button>
      <button id="pingIP" class="btn btn-primary">Ping IP</button>
      <div id="tableWrapper"/>
    </html>
    </panel>
  </row>
  <row depends="$htmlevent$">
    <panel>
      <html id="htmlevent">
      <label>eventdate Stuff do not delete</label>
        <div id="eventdata"/>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="snowfields">
        <div id="snowcont">
          <br/>
          <input type="radio" id="optinc" name="snowtype" value="Incident"/>
          <label for="incident">Incident</label>
          <input type="radio" id="optcase" name="snowtype" value="Case"/>
          <label for="case">Case</label>
          <label for="fsev">Severity:</label>
          <input type="text" id="sevfield" name="fsev"/>
          <label for="fshortdesc">Short Description:</label>
          <input type="text" id="shortdesc" name="fshortdesc"/>
          <label for="fdescription">Description:</label>
          <input type="text" id="descriptiontext" name="fdescription"/>
          <label for="fgroup">Group:</label>
          <select name="fgroup" id="assigngroup">
            <option value="noc">Telecoms Network Operations Centre</option>
            <option value="support">Telecoms Network Operations Centre Support Admin</option>
          </select>
          <label for="fcat">Category:</label>
          <select name="fcat" id="cat">
            <option value="none"/>
            <option value="datanetwork">Data Network</option>
            <option value="transmission">Transmission Network</option>
            <option value="infrastructure">Infrastructure</option>
            <option value="networkmanagement">Network Management</option>
            <option value="thirdparty">Third Party</option>
            <option value="itsystems">IT Systems</option>
            <option value="circuit">Circuit</option>
            <option value="security">Security</option>
            <option value="sites">Sites</option>
            <option value="controlsystems">Control Systems</option>
            <option value="mobile">Mobile</option>
            <option value="video">Video</option>
            <option value="voicerec">Voice Recording</option>
            <option value="voice">Voice</option>
            <option value="cti">CTI</option>
            <option value="reporting">Reporting</option>
          </select>
          <label for="fsubcat">Sub Category:</label>
          <select name="fsubcat" id="subcat">
           <option value="none"/>
          </select>
          <label for="fcasetype">Case Fault Type:</label>
          <select name="fcasetype" id="casefaulttype">
            <option value="none">--None--</option>
            <option value="servicedown">Service Down</option>
            <option value="degradedserv">Degraded Service</option>
            <option value="rfo">RFO</option>
            <option value="doa">DOA</option>
            <option value="telephony">Telephony</option>
          </select>
          <label for="fsla">SLA:</label>
          <select name="fsla" id="sla">
            
          </select>
          <label for="fcmdb">CMDB CI:</label>
          <input type="text" id="cmdbci" name="fcmdb"/>
          <br/>
          <button id="submitsnow" class="btn btn-primary">Submit</button>
          <button id="cancelsnow" class="btn btn-primary">Cancel</button>
        </div>
      </html>
    </panel>
  </row>
</form>

 

 

 

"Leaky xml"

 

<form script="index.js" stylesheet="style.css" hideFilters="true">
  <label>NOC Radio New Alarms</label>
  <search id="dataTableSearch">
    <query>| inputlookup noc_alerts_kv  | search State=New OR State=Closing $OpenState$ | search $datasource$ | search $severity$  | eval AlertText = if(isnull(AlertText), Object, Hostname + " " + AlertText) | search AlertText = $textsearch$  | eval Key = _key  | table Key DateTime Hostname IP_Address SnowID Severity Datasource State CustomerID AlertText Object | $sort$</query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <refresh>$refreshrate$</refresh>
    <sampleRatio>1</sampleRatio>
  </search>
  <fieldset autoRun="true" submitButton="true">
    <input type="dropdown" token="font">
      <label>Fontsize</label>
      <choice value="x-small">x-small</choice>
      <choice value="small">small</choice>
      <choice value="medium">medium</choice>
      <choice value="large">large</choice>
      <choice value="x-large">x-large</choice>
      <choice value="xx-large">xx-large</choice>
      <choice value="xxx-large">xxx-large</choice>
      <default>medium</default>
      <initialValue>medium</initialValue>
    </input>
    <input type="multiselect" token="fields">
      <label>Show Columns</label>
      <default>AlertText,Datasource,DateTime,SnowID</default>
      <initialValue>AlertText,Datasource,DateTime,SnowID</initialValue>
      <fieldForLabel>Fields</fieldForLabel>
      <fieldForValue>Fields</fieldForValue>
      <search>
        <query>| inputlookup AlertFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter> </delimiter>
    </input>
    <input type="dropdown" token="sort" searchWhenChanged="true">
      <label>Default Sort</label>
      <default>sort - DateTime</default>
      <initialValue>sort - DateTime</initialValue>
      <fieldForLabel>SortField</fieldForLabel>
      <fieldForValue>SortQuery</fieldForValue>
      <search>
        <query>| inputlookup SortFields.csv</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="dropdown" token="refreshrate">
      <label>Refresh Rate</label>
      <choice value="10">10 secs</choice>
      <choice value="15">15 secs</choice>
      <choice value="20">20 secs</choice>
      <choice value="30">30 secs</choice>
      <choice value="45">45 secs</choice>
      <choice value="60">1 min</choice>
      <choice value="120">2 mins</choice>
      <choice value="300">5 mins</choice>
      <choice value="3600">Hourly</choice>
      <choice value="86400">Daily</choice>
      <default>30</default>
      <initialValue>30</initialValue>
    </input>
    <input type="multiselect" token="severity">
      <label>Severity</label>
      <valuePrefix>Severity =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DropdownSev</fieldForLabel>
      <fieldForValue>DropdownSev</fieldForValue>
      <search>
        <query>| inputlookup DropdownSev.csv | dedup DropdownSev</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="multiselect" token="datasource">
      <label>Sources</label>
      <valuePrefix>Datasource =</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>DTSource</fieldForLabel>
      <fieldForValue>DTSource</fieldForValue>
      <search>
        <query>| inputlookup DatasourceNOC.csv | dedup DTSource | sort DTSource</query>
        <earliest>-15m</earliest>
        <latest>now</latest>
      </search>
      <default>MDSRadio,Siaeradio</default>
      <initialValue>Siaeradio,MDSRadio</initialValue>
    </input>
    <input type="checkbox" token="OpenState">
      <label></label>
      <choice value="*">Include Acknowledged</choice>
      <change>
        <condition label="Include Acknowledged">
          <set token="OpenState">OR State=Acknowledged</set>
        </condition>
        <condition>
          <set token="OpenState"></set>
        </condition>
      </change>
      <delimiter> </delimiter>
    </input>
    <input type="text" token="textsearch" searchWhenChanged="false">
      <label>Freeform Text</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <html>
      <img src="/static/app/noc/images/xxx.jpg"  style="width:359px; height:85px;"/>
  </html>
    </panel>
  </row>
  <row>
    <panel>
      <html>
        
      <div id="tableWrapper"/>
      <div id="selectedrowcount"/>
      <button id="selectall" class="btn btn-primary">Select All</button>
      <button id="unselectall" class="btn btn-primary">Unselect All</button>
      <button id="acknow" class="btn btn-primary">Acknowledge</button>
      <button id="close" class="btn btn-primary">Close</button>
      <button id="submitInc" class="btn btn-primary">Add SNOW No.</button>
      <button id="raiseInc" class="btn btn-primary">Raise SNOW</button>
      <button id="getEventQuery" class="btn btn-primary">Get Event Data</button>
      <button id="clearEventQuery" class="btn btn-primary">Clear Event Data</button>
      <button id="openIPBrowser" class="btn btn-primary">Open in Browser</button>
      <button id="pingIP" class="btn btn-primary">Ping IP</button>

    </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="htmlevent">
        <div id="eventdata"/>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <html id="snowfields">
        <div id="snowcont"><br />
          <input type="radio" id="optinc" name="snowtype" value="Incident" />
          <label for="incident">Incident</label>
          <input type="radio" id="optcase" name="snowtype" value="Case"  />
          <label for="case">Case</label>
          <label for="fsev">Severity:</label>
          <input type="text" id="sevfield" name="fsev"  />
          <label for="fshortdesc">Short Description:</label>
          <input type="text" id="shortdesc" name="fshortdesc"  />
          <label for="fdescription">Description:</label>
          <input type="text" id="descriptiontext" name="fdescription" />
          <label for="fgroup">Group:</label>
          <select name="fgroup" id="assigngroup">
            <option value="noc">Telecoms Network Operations Centre</option>
            <option value="support">Telecoms Network Operations Centre Support Admin</option>
          </select>
          <label for="fcat">Category:</label>
          <select name="fcat" id="cat">
            <option value="none"></option>
            <option value="datanetwork">Data Network</option>
            <option value="transmission">Transmission Network</option>
            <option value="infrastructure">Infrastructure</option>
            <option value="networkmanagement">Network Management</option>
            <option value="thirdparty">Third Party</option>
            <option value="itsystems">IT Systems</option>
            <option value="circuit">Circuit</option>
            <option value="security">Security</option>
            <option value="sites">Sites</option>
            <option value="controlsystems">Control Systems</option>
            <option value="mobile">Mobile</option>
            <option value="video">Video</option>
            <option value="voicerec">Voice Recording</option>
            <option value="voice">Voice</option>
            <option value="cti">CTI</option>
            <option value="reporting">Reporting</option>
          </select>
          <label for="fsubcat">Sub Category:</label>
          <select name="fsubcat" id="subcat">
           <option value="none"></option>
          </select>
          <label for="fcasetype">Case Fault Type:</label>
          <select name="fcasetype" id="casefaulttype">
            <option value="none">--None--</option>
            <option value="servicedown">Service Down</option>
            <option value="degradedserv">Degraded Service</option>
            <option value="rfo">RFO</option>
            <option value="doa">DOA</option>
            <option value="telephony">Telephony</option>
          </select>
          <label for="fsla">SLA:</label>
          <select name="fsla" id="sla">
            
          </select>
          <label for="fcmdb">CMDB CI:</label>
          <input type="text" id="cmdbci" name="fcmdb"  /><br />
          <button id="submitsnow" class="btn btn-primary">Submit</button>
          <button id="cancelsnow" class="btn btn-primary">Cancel</button>
        </div>
      </html>
    </panel>
  </row>
</form>

 

Anyone got any ideas?

Labels (1)

mscomms
Path Finder

The issue appears to occur when you hide columns using the .column(i).visible(false) method as soon as you hide a column the DataTable starts leaking memory I have worked around this using CSS to hide the columns

0 Karma

mscomms
Path Finder

Update

If you create a new empty dashboard and make a trivial change to it then save it, then edit it and paste in the leaky xml the dashboard doesn't leak

 

ivan_mirosav
Explorer

Hi mscomms,

I'm experiencing much the same behavior from a memory standpoint. Can you show me the search you used to identify the leaky dashboard?

 

0 Karma

ivan_mirosav
Explorer

Very useful! I see now that you mentioned this is eating PC memory. My issue is similar in behavior but in regards to Search Head memory. I'll keep this tool in mind for browser memory consumption on user PCs.

Thanks!

0 Karma

mscomms
Path Finder

Hi Ivan, 

I opened the dashboard as normal then hit F12 to open DevTools, go to the Memory tab and hit start in the bottom left of the screen you will see a page like this

mscomms_0-1617800481506.png

 

let it run and you will see peaks of memory useage when the dashboard runs its query. Immediatly after running the peaks will be filled with a colour (in my case blue) which should clear as the dashboard reruns its query. Ig the spikes stay full yopu are leaking memory

Thisd is my non leaky dashboard

mscomms_1-1617800751367.png

 

and this is the leaky one

mscomms_2-1617800870197.png

 

once it has captured some data click the red button top left and let the capture process, once it has completed processing you can inspect what is using the memory which will hopefully give you and idea where to investigate.

 

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...