Getting Data In

Why are Total events not matching with breakdown searches?

hperez
Explorer

Hello,

I'm creating a visualization and attempting to show the total amount of events, and break them down by a specific field. 

So my initial search would be something like the search below, where I just count all the events for a specific sourcetype.

index=foo sourcetype=bar | stats count as "Total Events for Security Control"

The other searches would filter these by evaluating a third field and counting the ones that are true for the condition and the ones that are not.

index=foo sourcetype=bar baz="Blocked" | stats count as "Total Blocked"

index=foo sourcetype=bar baz!="Blocked" | stats count as "Total Blocked"

The issue that I'm seeing is that for one of my sourcetype, the total number of events is not equal to the sum of the breakdown searches. Any idea as to why this might be happening?

Labels (2)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If any event is missing the "baz" field then it will not be counted in either breakdown search so the sum of the breakdown counts will not match the Total count.  Use this search to allow for when baz is null.

index=foo sourcetype=bar NOT baz="Blocked" 
| stats count as "Total Not Blocked"
---
If this reply helps you, Karma would be appreciated.

View solution in original post

bowesmana
SplunkTrust
SplunkTrust

What @richgalloway says, but as an additional point, it sounds like this may be a candidate for using a base search in the dashboard, which will help improve your dashboard speed.

If you're not familiar with base searches, see here for 'Post Process searches' https://docs.splunk.com/Documentation/Splunk/9.1.0/Viz/Savedsearches

In your XML you might have something like this

<search id="base">
  <query>
index=foo sourcetype=bar
| stats count by baz
  </query>
</search>

then for the post process searches you could do these 3 searches, one for each of the visualisations you want

<search base="base">
  <query>
| stats sum(count) as Total
  </query>
</search>

<search base="base">
  <query>
| where baz="Blocked"
| stats sum(count) as Blocked
  </query>
</search>

<search base="base">
  <query>
| where baz!="Blocked"
| stats sum(count) as Not_Blocked
  </query>
</search>

Hope this is useful

hperez
Explorer

I wasn't aware of this capability. Thank you. I'll take a look and see if I can implement it.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If any event is missing the "baz" field then it will not be counted in either breakdown search so the sum of the breakdown counts will not match the Total count.  Use this search to allow for when baz is null.

index=foo sourcetype=bar NOT baz="Blocked" 
| stats count as "Total Not Blocked"
---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...