advanced dashboard canceling search?

[caphrim007]

I have a view with 2 blocks (similar but different in layout) on the page that look like this.

  <module name="NullModule" layoutPanel="panel_row1_col1">
    <module name="GenericHeader" layoutPanel="panel_row1_col2">
      <param name="label">URLs inserted into savory</param>
    </module>
    <module name="TimeRangePicker" layoutPanel="panel_row1_col2">
      <param name="default">Last 24 hours</param>
      <param name="searchWhenChanged">True</param>

      <module name="NullModule"/>
      <module name="HiddenSearch" autoRun="True">
        <param name="search">sourcetype="savory-messages" splunk_server="host.site.org" AND ("Added new entry" OR "Added new URL")</param>
        <module name="HiddenPostProcess">
          <param name="search">timechart span="1h" count(_raw)</param>
          <module name="JobProgressIndicator">
            <module name="HiddenChartFormatter">
              <param name="chart">column</param>
              <param name="primaryAxisTitle.text">Time</param>
              <param name="secondaryAxisTitle.text">Number of URLs</param>
              <param name="legend.placement">none</param>
              <module name="FlashChart" layoutPanel="panel_row1_col2">
                <param name="width">100%</param>
                <param name="height">400px</param>
              </module>
            </module>
          </module>
        </module>

        <module name="HiddenPostProcess" layoutPanel="panel_row1_col2">
          <param name="search">timechart span="1h" count(_raw)</param>
          <module name="JobProgressIndicator">
            <module name="SimpleResultsTable">
              <param name="count">24</param>
            </module>
          </module>
        </module>
      </module>
    </module>
  </module>

When the page loads, I get results in my chart and table, but they are incomplete. If I change the timerange using the picker, I get more incomplete.

For instance, if I select "24 hours" I get results in the chart and table that go back to approximately noon today. If I select "7 days" I get data that goes back to approximately sept 16th.

If I manually run any of the searches, I get all the results.

When I used firebug's net tab and the time picker, I see a POST jobs followed by a POST control which includes an action=cancel.

I don't know if that is the culprit, but I can't explain why it wouldn't be receiving all the results as it should.

Ideas?

[sideview · Accepted solution]

It's not anything being cancelled, it's just a limitation of postProcess that you have to be careful with.

First though, there's a higher order problem with this view. Lets take care of that first.

1) If you picture the data flowing down and inward through the modules, searches get dispatched as soon as the data hits a module that needs the search to be running. So for example HiddenSearch, HiddenPostProcess, TimeRangePicker etc.. -- these modules do not require a running search so nothing will get kicked off. On the other hand JobProgressIndicator and SimpleResultsTable need the search to be running or else they're meaningless so if the package of search data hits those points and the search hasnt been kicked off already, it'll get kicked off there.

Given that info, take another look at your XML and you'll see that even though you have two HiddenPostProcesses, the searches wont get dispatched until the data is in the subtrees-- so you're not getting the benefit of HiddenPostProcess and you're actually going to have two searches running.

What you want instead is probably a single JobProgressIndicator that's higher up in the hierarchy, and that has both HiddenPostProcess modules downstream.

2) OK. The deal with postProcess is that it cant work with what it doesnt have. Splunk wont keep every _raw value of every event around -- in fact it will only keep 50,000 or so.

So the timechart commands on your HiddenPostProcess are going to be operating on incomplete input. The answer is simple although unintuitive-- change the base search to be able to provide complete information to both.

So for example, if you wanted a single search to feed both | timechart span="1h" count as well as something like: | chart sum(session_length) by username, you'd do this:

base search: <your search> | bin _time span="1h" | stats count by session_length, user, _time

postProcess1: timechart span="1h" sum(count) as count

postProcess2: chart sum(session_length) by username

This technique is often referred to as the 'datacube' technique, cause you're building a higher dimensional cube of data, and then using postProcess to draw projections from that cube to create the end reports.

There's a page in the docs about this -- http://www.splunk.com/base/Documentation/4.1.5/Developer/PostProcess
and if you prefer hands-on examples to tinker with yourself, download the app on splunkbase called "UI examples for 4.1". It has a view discussing techniques and pitfalls around datacubes and postProcess.

Note in particular that you have to worry about the rows outputted by the stats command, where 'count' may be >1. this is what the sum(count) as count is doing in the timechart command.

[caphrim007]

Out of curiosity, I removed the HiddenPostProcess modules from each section and replaced them with regular searches and the charts and tables appear to be correct.

What would cause this? This is really suboptimal for this particular use case since I'll need to run 4 searches now and would prefer to stay with 2.

Any help, including pointing to (non-obvious) documentation, would be appreciated.