HIddenPostProcess narrow time range

[christopherutz]

I have a dashboard that uses HiddenSearch and HiddenPostProcess to show 4 line charts. The first two look at 3 monthes worth of events and show the avg and min of a field respectively. The last 2 charts should show 1 monthes worth of data for the same two statistics. I was hoping I could run the HiddenSearch for three monthes worth of data and then have the two 1 month HiddenPostProcess do something like this:

<module name="HiddenSearch" layoutPanel="panel_row1_col1" autoRun="True">
  <param name="search"> sourcetype=perfdata End | fields _time, runtime, class</param>
  <param name="earliest">-2160h</param>

<module name="HiddenPostProcess" layoutPanel="panel_row1_col1">
  <param name="search">search earlist=-720h | timechart avg(runtime) by class</param>
    <module name="HiddenChartFormatter">
      <param name="chart">line</param>
      <param name="chartTitle">Average Runtime 1 Month</param>
      <param name="primaryAxisTitle.text">time</param>
      <param name="secondaryAxisTitle.text">Average Runtime</param>
      <param name="legend.placement">right</param>
        <module name="FlashChart">
          <param name="width">100%</param>
          <param name="height">350px</param>
        </module>
    </module>
</module>

This does not appear to work. Is this possible, or do I need to run a hiddensearch for each timerange?

[sideview · Accepted solution]

This can be done.
First note that because you've set it up such that the HiddenSearch contains a search for events, you are subject to various limitations of postProcess. Most notably this will not work correctly if there are ever more than 10,000 events in that underlying search.

But this search language:

<your search> | eval isMostRecentMonth=if(_time>relative_time(now(), "-1mon"), 1, 0)

will add a field called isMostRecentMonth. For the events in the last month it will be 1. For the others it'll be 0.

A) If you know the limitations of postProcess will never be an issue, you can do this:

base search:
<your search> | eval isMostRecentMonth=if(_time>relative_time(now(), "-1mon"), 1, 0) | fields _time, runtime, class, isMostRecentMonth

postprocess 1 :
timechart avg(runtime) by class

postprocess 2:
search isMostRecentMonth=1 | timechart avg(runtime) by class

B) And if you think you'd better protect yourself against the limitations, the 'avg' case gets pretty nasty looking but do this.

base search:
<your search> | eval isMostRecentMonth=if(_time>relative_time(now(), "-1mon"), 1, 0) | bucket _time span=1d | stats count by _time, runtime, class, isMostRecentMonth

postprocess 1 :
timechart sum(eval(count*runtime)) as total_runtime by class

postprocess 2:
search isMostRecentMonth=1 | timechart sum(eval(count*runtime)) as total_runtime by class

(This is a pretty complex case though. For further reading check out the ui examples app, and the view called 'using postProcess on dashboards'. )

[sideview]

it doesnt make much sense why we did it this way, but the "events" are limited at 10K (or possibly 50K I forget), but as of 4.1.4 or so the reporting commands like stats will actually output ALL rows. So the postProcess limitation where the rows get truncated only applies when it's literally events that make up the rows.

[christopherutz]

After walking through the queries you suggested it is making sense, however I still have more than 10000 bucketed stats. The cross of runtime and class is quite large. My runtime is in ms and I think I have some freedom as far as the precision I care about. I am pretty confident I can make this work by rounding runtime to 100 ms of precision.

[sideview]

Try it out by adding each command from the postProcess one at a time to the main search look at the output in the searchView. It'll make sense. Sometimes we refer to this as the 'datacube' approach, because you can hang all kinds of different postProcess searches off of it, just by adding another field or two to the stats command.

[christopherutz]

Thanks Nick. I am close to a million events so I do need worry about the limitations. I understand what all the searches are doing but I am not convinced they are mathematically equivalent. It's possible I just need to think about it in more detail.