Trigger alert when rolled up dashboard color goes ...

matthewwhittle · ‎11-06-2020

Hi all,

(refer to the screenshot attached)

I have a dashboard with 30 or so panels just like the one below. They feature a search as well as a green dot if the numbers in the last hour are within acceptable ranges.

Looking at the dashboard html below, you can see the MainTrafficSearch causes the color of the svg circle on the row to change colors.

It also updates a color named V7Color. This is the main color for the entire dashboard. It represents the worst color of any of the searches in the dashboard.

Now what I want is to configure an alert so that if the color of the V7Color goes to yellow or red, I send out an email. I would like to not have to do a massive search query which is the combination of all 30 of my searches. If possible, I would like to not have to configure 30 separate alerts (because the app is so tightly monitored that many times, when one part of the app goes bad, several do).

I also already have basically the same search in two places, if possible, I'd like to not have to write the search in a third place. (Unless it is possible to write the search one place and reference it as a variable in all the other places)

I understand that what I am hoping to do might not be possible. But if you were in my shoes, what route would you take? Thanks

  <search id="MainTrafficSearch">
    <query>sourcetype=provider_api_jidp env=$envselection$ index=$envselection$ "AccessTokenInterceptor" | rex field=_raw "ajp-/(?&lt;ipthread&gt;[0-9\.\-:]+)" | rex field=_raw "(GET|POST|PUT|DELETE|OPTIONS) (?&lt;resource&gt;[a-zA-Z\.]+ \(.*\)) " | dedup _time host resource ipthread | stats count | eval status=case(count &gt; 500000, 2, count &gt; 250000, 1, count &lt; 100, 1, count &lt; 10, 2, true(), 0) | table status</query>
    <earliest>-60m</earliest>
          <latest>$timepicker.latest$</latest>
    <progress>
      <eval token="MainTrafficColor">case($result.status$=="0","green",$result.status$=="2","red",$result.status$=="1","yellow",true(),"grey")</eval>
      <eval token="V7Color">case($result.status$=="2","red",$result.status$=="1" AND $V7Color$ != "red","yellow",true(),$V7Color$)</eval>
    </progress>
  </search>


…

  <row>
    <panel>
      <title>Traffic</title>
      <html>
        <div style="width:100%;height:30px;position:absolute">
         <svg xmlns="http://www.w3.org/2000/svg" style="margin-left:95%">
         <circle cx="13" cy="13" r="10" fill="$MainTrafficColor$"/>
        </svg>
        </div>
        </html>
      <single>
        <title>Number of Calls</title>
        <search>
          <query>sourcetype=provider_api_jidp index=$envselection$ env=$envselection$ "AccessTokenInterceptor" | rex field=_raw "ajp-/(?&lt;ipthread&gt;[0-9\.\-:]+)" | rex field=_raw "(GET|POST|PUT|DELETE|OPTIONS) (?&lt;resource&gt;[a-zA-Z\.]+ \(.*\)) " | dedup _time host resource ipthread | stats count</query>
          <earliest>$timepicker.earliest$</earliest>
          <latest>$timepicker.latest$</latest>
        </search>
        <option name="colorMode">block</option>
        <option name="drilldown">none</option>
        <option name="height">50</option>
        <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
        <option name="refresh.display">progressbar</option>
        <option name="unitPosition">before</option>
        <option name="useColors">0</option>
        <option name="useThousandSeparators">1</option>
      </single>
    </panel>
  </row>

Trigger alert when rolled up dashboard color goes red

panel

.conf24 | Registration Open!

ICYMI - Check out the latest releases of Splunk Edge Processor

Introducing the 2024 SplunkTrust!