Splunk Search

post-process not delivering all events to chart?

Path Finder

I have the following xml

<module name="HiddenSearch" layoutPanel="panel_row2_col1" group="XXX" autoRun="True">
<param name="search">apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") </param>
<param name="earliest">-30m@m</param>
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp1">
<param name="search">timechart count by status</param>
<module name="HiddenChartFormatter">
<param name="charting.chart">column</param>
<param name="charting.chart.stackMode">stacked</param>
<param name="charting.legend.placement">bottom</param>
<module name="JobProgressIndicator"/>
<module name="FlashChart">
<param name="width">100%</param>
<param name="enableResize">True</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
</module>
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp2">
<param name="search">top responseStatus | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100</param>
<module name="HiddenChartFormatter">
<param name="charting.chart">fillerGauge</param>
<param name="charting.chart.style">shiny</param>
<param name="charting.chart.orientation">x</param>
<param name="charting.chart.usePercentageRange">true</param>
<param name="charting.chart.usePercentageValue">true</param>
<module name="JobProgressIndicator"/>
<module name="FlashChart">
<param name="width">100%</param>
<param name="enableResize">False</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
</module>

What happens is that the gauge from TOP is correctly displayed, but the timechart only gets 5 minutes of data. It should be displaying all 30 minutes of data. Any suggestions?

ETA: If you change the timechart to chart count by _time then the chart will only display a 5 minute graph, vs a 30 minute graph with 5 minutes of data with timechart.

1 Solution

SplunkTrust
SplunkTrust

You should check out the docs around postProcess. In particular if the base search does not contain any transforming search commands splunk will not preserve full information about the events past the 50,000'th event.

Check out the UI Examples app on Splunkbase, and read the page called "Using postProcess on dashboards".

or check out the docs here. http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

The answer in your case is as follows:

instead of having this search

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR")

You want to have this base search:

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") | bin _time span="1min" | stats count by _time responseStatus

Splunk basically will not keep arbitrarily large numbers of events around. On the other hand if the search has transforming commands, it will assemble a complete result set and not cut any corners. Adding these bin and stats commands means that you'll have a much more compact and efficient data set to work with, with no missing information.

View solution in original post

SplunkTrust
SplunkTrust

You should check out the docs around postProcess. In particular if the base search does not contain any transforming search commands splunk will not preserve full information about the events past the 50,000'th event.

Check out the UI Examples app on Splunkbase, and read the page called "Using postProcess on dashboards".

or check out the docs here. http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

The answer in your case is as follows:

instead of having this search

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR")

You want to have this base search:

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") | bin _time span="1min" | stats count by _time responseStatus

Splunk basically will not keep arbitrarily large numbers of events around. On the other hand if the search has transforming commands, it will assemble a complete result set and not cut any corners. Adding these bin and stats commands means that you'll have a much more compact and efficient data set to work with, with no missing information.

View solution in original post

Path Finder

This worked! The only downside to this is I had to do some trickery to get results similar to the top command.

eventstats sum(hitcount) as totalCount | eventstats sum(hitcount) as statusCount by responseStatus | dedup responseStatus | eval percent=(statusCount/totalCount)*100 | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100

0 Karma

Splunk Employee
Splunk Employee

I think you'll want to switch your top & your where so that you are getting a top of ERRORs instead of subset of ERRORs from whatever topped.

from

<param name="search">top responseStatus | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100</param>

to

<param name="search">where match(responseStatus,"ERROR") | top responseStatus | gauge percent 0 5 10 100</param>
0 Karma

Splunk Employee
Splunk Employee

Yeah, I misread the problem/question

0 Karma

Path Finder

Unless you think that this is affecting the timechart postprocess, the change you are suggesting doesn't help. I only want the subset of errors.

0 Karma