Splunk Search

Returned Results of "0" when event count > 0

willadams
Contributor

I am writing a custom dashboard that shows the number of alerts based on severity. The severities are Critical, High, Medium, Low and Info.

The problem in my dashboard is that for each severity the returned numbers are vastly different to the native search query (as an example, a "critical" entry I see a value of "0" when it has finished running, where a "low" entry has "600" when it has finished running. While the dashboard is generating I see a value of "3" appear (using the critical for example) before it finishes and displays a "0".

If I run my query in SPL directly, for example

index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category | where severity="critical" | stats count(severity) as count

I get a returned result of "3" where the dashboard shows me "0"

Doing the same thing for low,

index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category | where severity="low" | stats count(severity) as count

I get a returned result of "2000", whereas the dashboard shows me "600"

My dashboard uses a base search to try and optimise it (the number of events in the logs (over a 24 hour period) is around 600000 with the job taking close to 56 seconds to complete.

My dashboard is written as follows:

<form theme="darK" hideFilters="true" refresh="1800">
<label>Dashboard</label>
<description>A description</description>
<search id="base">
<query>
  index=firewall sourcetype="threat" NOT [|inputlookup exclusion_ip.csv | fields src] | fields severity, src, category
</query>
<fieldset submitButton="false">
    <input type="time" token="time_tok">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
<row>
<panel>
<single>
<title>Critical</title>
<search base="base">
<query>
  | where severity="critical"
  | stats count(severity) as count
</query>
</search>
</single>
</panel>
<panel>
<single>
<title>High</title>
<search base="base">
<query>
  | where severity="high"
  | stats count(severity) as count
</query>
</search>
</single>
</panel>
</row>
..................

 <row>
  <panel>
      <table>
        <search base="base">
          <query>
              | search severity="critical"
              | stats count(severity) As Count by category
              | sort - Count
         </query>
        </search>
   </table>
  </panel>
</row>
    </form>

While the counts for every other panel does return a result, the number in the panels is different to the numbers of the native search.

I suspect this is a limits thing.....................

If I remove the "informational" part in my search which produced "110000" results and found that the remainder of my dashboard panels increased in number. The returned results for example is

"2" critical,
"193000" high
"9400" medium
"1400" low
"0" informational (obviously)

With the "informational" part added back into the search, my dashboard numbers change to

"0" critical
"89000" high
"6200" medium
"500" low
"110000" informational

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @willadams,
do you have the same results if you use a past time frame (e.g. yesterday or last hour) wiithou latest=now?

Ciao.
Giuseppe

0 Karma

willadams
Contributor

The logs have only been ingested since this morning so not a full 24 hours yet. The log technically started at aroind 3am my time and from when I wrote this question it has been 13 hours

0 Karma

willadams
Contributor

I will however try and shorten the time frame to see what occurs.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @willadams,
to reduce the time frame isn't important because you haven't relevant subsearches (only one inputlookup), but it could be relevant the last indexed events, for this reason try earliest=-3h@h latest=-2h@h

Ciao.
Giuseppe

0 Karma

willadams
Contributor

The time difference reduced numbers but was incorrect based on the calculations. As part of my dashboard (which ultimately has 10 panels) I opted to go a different route and have increased the performance of the search better. The dashboard now loads within 10 to 15 seconds vs the almost 60 seconds the "optimised" search ran.

I found that running the search manually in native SPL was significantly faster than the search I had configured above. Instead I created 5 seperate base search (one for each severity type) and then removed the "|search severity="..." from each panel.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...