topic Returned Results of "0" when event count > 0 in Splunk Search

Returned Results of "0" when event count > 0

willadams — Tue, 19 Nov 2019 08:00:57 GMT

I am writing a custom dashboard that shows the number of alerts based on severity. The severities are Critical, High, Medium, Low and Info.

The problem in my dashboard is that for each severity the returned numbers are vastly different to the native search query (as an example, a "critical" entry I see a value of "0" when it has finished running, where a "low" entry has "600" when it has finished running. While the dashboard is generating I see a value of "3" appear (using the critical for example) before it finishes and displays a "0".

If I run my query in SPL directly, for example

index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category | where severity="critical" | stats count(severity) as count

I get a returned result of "3" where the dashboard shows me "0"

Doing the same thing for low,

index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category | where severity="low" | stats count(severity) as count

I get a returned result of "2000", whereas the dashboard shows me "600"

My dashboard uses a base search to try and optimise it (the number of events in the logs (over a 24 hour period) is around 600000 with the job taking close to 56 seconds to complete.

My dashboard is written as follows:

<form theme="darK" hideFilters="true" refresh="1800">
<label>Dashboard</label>
<description>A description</description>
<search id="base">
<query>
  index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category
</query>
<fieldset submitButton="false">
    <input type="time" token="time_tok">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
<row>
<panel>
<single>
<title>Critical</title>
<search base="base">
<query>
  | where severity="critical"
  | stats count(severity) as count
</query>
</search>
</single>
</panel>
<panel>
<single>
<title>High</title>
<search base="base">
<query>
  | where severity="high"
  | stats count(severity) as count
</query>
</search>
</single>
</panel>
</row>
..................

 <row>
  <panel>
      <table>
        <search base="base">
          <query>
              | search severity="critical"
              | stats count(severity) As Count by category
              | sort - Count
         </query>
        </search>
   </table>
  </panel>
</row>
    </form>

While the counts for every other panel does return a result, the number in the panels is different to the numbers of the native search.

I suspect this is a limits thing.....................

If I remove the "informational" part in my search which produced "110000" results and found that the remainder of my dashboard panels increased in number. The returned results for example is

"2" critical,
"193000" high
"9400" medium
"1400" low
"0" informational (obviously)

With the "informational" part added back into the search, my dashboard numbers change to

"0" critical
"89000" high
"6200" medium
"500" low
"110000" informational

Re: Returned Results of "0" when event count > 0

gcusello — Tue, 19 Nov 2019 08:18:35 GMT

Hi @willadams,
do you have the same results if you use a past time frame (e.g. yesterday or last hour) wiithou latest=now?

Ciao.
Giuseppe

Re: Returned Results of "0" when event count > 0

willadams — Tue, 19 Nov 2019 08:26:49 GMT

The logs have only been ingested since this morning so not a full 24 hours yet. The log technically started at aroind 3am my time and from when I wrote this question it has been 13 hours

Re: Returned Results of "0" when event count > 0

willadams — Tue, 19 Nov 2019 08:27:13 GMT

I will however try and shorten the time frame to see what occurs.

Re: Returned Results of "0" when event count > 0

gcusello — Tue, 19 Nov 2019 08:43:30 GMT

Hi @willadams,
to reduce the time frame isn't important because you haven't relevant subsearches (only one inputlookup), but it could be relevant the last indexed events, for this reason try earliest=-3h@h latest=-2h@h

Ciao.
Giuseppe

Re: Returned Results of "0" when event count > 0

willadams — Wed, 20 Nov 2019 04:00:56 GMT

The time difference reduced numbers but was incorrect based on the calculations. As part of my dashboard (which ultimately has 10 panels) I opted to go a different route and have increased the performance of the search better. The dashboard now loads within 10 to 15 seconds vs the almost 60 seconds the "optimised" search ran.

I found that running the search manually in native SPL was significantly faster than the search I had configured above. Instead I created 5 seperate base search (one for each severity type) and then removed the "|search severity="..." from each panel.