Why are Total events not matching with breakdown searches?

[hperez]

Hello,

I'm creating a visualization and attempting to show the total amount of events, and break them down by a specific field. 

So my initial search would be something like the search below, where I just count all the events for a specific sourcetype.

index=foo sourcetype=bar | stats count as "Total Events for Security Control"

The other searches would filter these by evaluating a third field and counting the ones that are true for the condition and the ones that are not.

index=foo sourcetype=bar baz="Blocked" | stats count as "Total Blocked"

index=foo sourcetype=bar baz!="Blocked" | stats count as "Total Blocked"

The issue that I'm seeing is that for one of my sourcetype, the total number of events is not equal to the sum of the breakdown searches. Any idea as to why this might be happening?

[bowesmana]

What @richgalloway says, but as an additional point, it sounds like this may be a candidate for using a base search in the dashboard, which will help improve your dashboard speed.

If you're not familiar with base searches, see here for 'Post Process searches' https://docs.splunk.com/Documentation/Splunk/9.1.0/Viz/Savedsearches

In your XML you might have something like this

<search id="base">
  <query>
index=foo sourcetype=bar
| stats count by baz
  </query>
</search>

then for the post process searches you could do these 3 searches, one for each of the visualisations you want

<search base="base">
  <query>
| stats sum(count) as Total
  </query>
</search>

<search base="base">
  <query>
| where baz="Blocked"
| stats sum(count) as Blocked
  </query>
</search>

<search base="base">
  <query>
| where baz!="Blocked"
| stats sum(count) as Not_Blocked
  </query>
</search>

Hope this is useful

[hperez]

I wasn't aware of this capability. Thank you. I'll take a look and see if I can implement it.

[richgalloway · Accepted solution]

If any event is missing the "baz" field then it will not be counted in either breakdown search so the sum of the breakdown counts will not match the Total count.  Use this search to allow for when baz is null.

index=foo sourcetype=bar NOT baz="Blocked" 
| stats count as "Total Not Blocked"