Splunk Search

Event count variation between base and non-base search dashboard panels

DEADBEEF
Path Finder

I have a dashboard which counts the number of times a user performed an action.  I have 3 time frames (last 24h, 7d, 30d) and thought I would try using  three base searches since I do more transformation with each "set" of data.

For testing, I left my 3 original panels and wrote 3 new base searches and added new panels to use those.  I noticed that the results from my original searches and the base searches don't match up.  But if you click the magnifying glass, "Open in Search" then it matches the non-base search.  The 24h search matches the non-base but the other two (7d & 30d) don't and I have no idea why.

screenshot.png

My XML

 

<dashboard>
  <label>base_testing</label>
  <search id="base24h">
    <query>
      index=foo
      | fields _time file user
    </query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
  </search>
  <search id="base7d">
    <query>
      index=foo
      | fields _time file user
    </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>
  <search id="base30d">
    <query>
      index=foo
      | fields _time file user
    </query>
    <earliest>-30d@h</earliest>
    <latest>now</latest>
  </search>  

  <row>
    <panel>
      <title>24 hours</title>
      <table>
        <search>
          <query>index=foo 
| fields _time file user 
| stats dc(file) AS "File Count" by user 
| sort - "File Count" 
| head 20</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
    <panel>
      <title>7 days</title>
      <table>
        <search>
          <query>index=foo
| stats dc(file) AS "File Count" by user
| sort - "File Count"
| head 20</query>
          <earliest>-7d@h</earliest>
          <latest>now</latest>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
    <panel>
      <title>30 days</title>
      <table>
        <search>
          <query>index=foo
| stats dc(file) AS "File Count" by user
| sort - "File Count"
| head 20</query>
          <earliest>-30d@d</earliest>
          <latest>now</latest>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
<!-- BASE ROW -->
<!-- BASE ROW -->
<!-- BASE ROW -->
  <row>
    <panel>
      <title>BASE 24 hours</title>
      <table>
        <search base="base24h">
          <query>| stats dc(file) AS "File Count" by user | sort - "File Count" | head 20</query>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
    <panel>
      <title>BASE 7 days</title>
      <table>
        <search base="base7d">
          <query>| stats dc(file) AS "File Count" by user | sort - "File Count" | head 20</query>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
    <panel>
      <title>BASE 30 days</title>
      <table>
        <search base="base30d">
          <query>| stats dc(file) AS "File Count" by user | sort - "File Count" | head 20</query>
        </search>
        <option name="count">5</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
</dashboard>

 

 

Labels (1)
Tags (2)
0 Karma

robinsonalex88
Explorer

How many results are returned by the 7d and 30d base searches?  If it is in excess of 500,000 then Splunk will silently ignore all but the first 500k.  The documentation also recommends using a transforming search as your base search.

 

0 Karma

DEADBEEF
Path Finder

@robinsonalex88 - both the 7d and 30d non-base search panels are able to list the correct 20.  I'm curious why the base search wouldn't be able to.  Either way, 7d has 40k events, 30d has 100k.  When I click (open in search) on the panel using the base search, it will run the search and its values will be correct (differing from what it displays on the dashboard panel).

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...