Dashboards & Visualizations

HiddenPostProcess 10000 limit? Configurable? How to overcome? By summarising?

Explorer

Hi all,

I am having difficulty working around what I understand to be a hard limit of 10000 events for the HiddenPostProcess module. I am looking for a way to reduce the granularity of the events going across, but without success.

Example problem code:

<module name="HiddenSearch" autoRun="False" layoutPanel="mainSearchControls">
    <param name="search">index=analysisindex01
        eventtype="EVENTS01" OR  
        eventtype="EVENTS02" OR 
        eventtype="EVENTS03" OR 
        eventtype="EVENTS04" OR
        eventtype="EVENTS05" OR 
        eventtype="EVENTS06" OR 
        eventtype="EVENTS07" OR
        eventtype="EVENTS08"
        timeformat="%d/%m/%Y" 
        starttime=10/11/2010 
        searchtimespandays=5
        | fields eventtype, _time
    </param>
    ... stuff ...
    <module name="HiddenPostProcess" layoutPanel="panel_row2_col1">
                    <param name="search">timechart span="1d" count(eventtype) by eventtype</parm>

This will fail. It will produce results, but they will be capped to 10000.

If I do

    timeformat="%d/%m/%Y" 
    starttime=10/11/2010 
    searchtimespandays=5
    | bucket _time span=1h
    | fields eventtype, _time

in an attempt to summarize the events a bit, I only get 1 row, for 1 day, in the reports generated, and the numbers in it make no sense.

Is there something I am missing?

Any and all help is greatly appreciated.

Tags (1)
1 Solution

Splunk Employee
Splunk Employee

bucket doesn't actually summarize at all. bucket just discretizes the value, e.g. in your search, it means for each event, snap the value of _time to the previous hour boundary.

What you'd want to do is something like

timeformat="%d/%m/%Y" starttime=10/11/2010 searchtimespandays=5 | bucket _time span=1h | stats count by _time eventtype

That will give you 1 row per day per eventtype

then in your hidden post process, you can do

timechart span=1d sum(count) by eventtype

View solution in original post

Splunk Employee
Splunk Employee

bucket doesn't actually summarize at all. bucket just discretizes the value, e.g. in your search, it means for each event, snap the value of _time to the previous hour boundary.

What you'd want to do is something like

timeformat="%d/%m/%Y" starttime=10/11/2010 searchtimespandays=5 | bucket _time span=1h | stats count by _time eventtype

That will give you 1 row per day per eventtype

then in your hidden post process, you can do

timechart span=1d sum(count) by eventtype

View solution in original post