HiddenPostProcess 10000 limit? Configurable? How to overcome? By summarising?

[Alex_Megremis]

Hi all,

I am having difficulty working around what I understand to be a hard limit of 10000 events for the HiddenPostProcess module. I am looking for a way to reduce the granularity of the events going across, but without success.

Example problem code:

<module name="HiddenSearch" autoRun="False" layoutPanel="mainSearchControls">
    <param name="search">index=analysisindex01
        eventtype="EVENTS01" OR  
        eventtype="EVENTS02" OR 
        eventtype="EVENTS03" OR 
        eventtype="EVENTS04" OR
        eventtype="EVENTS05" OR 
        eventtype="EVENTS06" OR 
        eventtype="EVENTS07" OR
        eventtype="EVENTS08"
        timeformat="%d/%m/%Y" 
        starttime=10/11/2010 
        searchtimespandays=5
        | fields eventtype, _time
    </param>
    ... stuff ...
    <module name="HiddenPostProcess" layoutPanel="panel_row2_col1">
                    <param name="search">timechart span="1d" count(eventtype) by eventtype</parm>

This will fail. It will produce results, but they will be capped to 10000.

If I do

    timeformat="%d/%m/%Y" 
    starttime=10/11/2010 
    searchtimespandays=5
    | bucket _time span=1h
    | fields eventtype, _time

in an attempt to summarize the events a bit, I only get 1 row, for 1 day, in the reports generated, and the numbers in it make no sense.

Is there something I am missing?

Any and all help is greatly appreciated.

[steveyz · Accepted solution]

bucket doesn't actually summarize at all. bucket just discretizes the value, e.g. in your search, it means for each event, snap the value of _time to the previous hour boundary.

What you'd want to do is something like

timeformat="%d/%m/%Y" starttime=10/11/2010 searchtimespandays=5 | bucket _time span=1h | stats count by _time eventtype

That will give you 1 row per day per eventtype

then in your hidden post process, you can do

timechart span=1d sum(count) by eventtype