Dashboard performance on load

leandromatperei — Tue, 17 Dec 2019 00:43:59 GMT

Expensive,

I have an example dashboard like this below with 12 queries that make up my dashboard, it loads new information every minute.

Thinking about the optimization scenario, it takes a while to load, is there any way to make it faster?

Maybe decreasing the number of queries will solve, but I need to keep 12 separate values.

<dashboard>
  <label>TESTE</label>
  <row>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 1</title>
        <search>
          <query>index=_internal  clientip="127.0.0.1" | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[5000]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 2</title>
        <search>
          <query>index=_internal  component=Metrics | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 3</title>
        <search>
          <query>index=_internal  file=jobs | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[4000]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 4</title>
        <search>
          <query>index=_internal  status=200 | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 5</title>
        <search>
          <query>index=_internal   component=PeriodicHealthReporter | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 6</title>
        <search>
          <query>index=_internal   user="splunk-system-user" | stats  count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 7</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 8</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 9</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 10</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 11</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
    <panel>
      <single depends="$showsinglevalue$">
        <title>TESTE 12</title>
        <search>
          <query>index=_internal | stats count as sourcetype</query>
          <earliest>$tempo.earliest$</earliest>
          <latest>$tempo.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="rangeColors">["0x53a051","0xdc4e41"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">0</option>
      </single>
    </panel>
  </row>
</dashboard>

Re: Dashboard performance on load

richgalloway — Tue, 17 Dec 2019 01:18:28 GMT

Are the last 6 panels supposed to be identical?

Re: Dashboard performance on load

leandromatperei — Tue, 17 Dec 2019 01:49:48 GMT

They are just to exemplify, may consider as different values.

May consider as 12 different values.

Re: Dashboard performance on load

gcusello — Tue, 17 Dec 2019 07:53:34 GMT

Hi @leandromatpereira,
at first you could use a Post Process Search, in other words, you could have a main search (index=_internal) and in each panel take the results of this search and add the other filters and calculations (e.g. | search clientip="127.0.0.1" | stats count as sourcetype).
You can see how to do this on Splunk Dashboard Examples App ( https://splunkbase.splunk.com/app/1603/ ) or at https://docs.splunk.com/Documentation/Splunk/8.0.0/Viz/Savedsearches#Post-process_searches_2 .

Then I don't understand the depends in each panel: where's the condition? but probably there a part of your dashboard that you didn't inserted in the question.

Then I see that some panels seem to be equal, maybe you could optimize them and reduce the number of panels.

Then the refresh every minute: is it mandatory? what's the execution time of your searches, have you time to reload?, maybe it could be possible to enlarge the refresh time.

At lease, remember that every search takes a CPU and didn't release it until it's working, this means that, when you run this dashboard, you're using 12 CPUs on Search Head and 12 on Indexers!
I don't know your architecture, but if two o three people use this dashboard you risk to block your system!

Ciao.
Giuseppe

topic Re: Dashboard performance on load in Dashboards & Visualizations

Dashboard performance on load

Re: Dashboard performance on load

Re: Dashboard performance on load

Re: Dashboard performance on load