Splunk Search

post-process not delivering all events to chart?

sf_user_199
Path Finder

I have the following xml

<module name="HiddenSearch" layoutPanel="panel_row2_col1" group="XXX" autoRun="True">
<param name="search">apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") </param>
<param name="earliest">-30m@m</param>
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp1">
<param name="search">timechart count by status</param>
<module name="HiddenChartFormatter">
<param name="charting.chart">column</param>
<param name="charting.chart.stackMode">stacked</param>
<param name="charting.legend.placement">bottom</param>
<module name="JobProgressIndicator"/>
<module name="FlashChart">
<param name="width">100%</param>
<param name="enableResize">True</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
</module>
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1_grp2">
<param name="search">top responseStatus | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100</param>
<module name="HiddenChartFormatter">
<param name="charting.chart">fillerGauge</param>
<param name="charting.chart.style">shiny</param>
<param name="charting.chart.orientation">x</param>
<param name="charting.chart.usePercentageRange">true</param>
<param name="charting.chart.usePercentageValue">true</param>
<module name="JobProgressIndicator"/>
<module name="FlashChart">
<param name="width">100%</param>
<param name="enableResize">False</param>
<module name="ConvertToDrilldownSearch">
<module name="ViewRedirector">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
<module name="ViewRedirectorLink">
<param name="viewTarget">flashtimeline</param>
</module>
</module>
</module>
</module>

What happens is that the gauge from TOP is correctly displayed, but the timechart only gets 5 minutes of data. It should be displaying all 30 minutes of data. Any suggestions?

ETA: If you change the timechart to chart count by _time then the chart will only display a 5 minute graph, vs a 30 minute graph with 5 minutes of data with timechart.

1 Solution

sideview
SplunkTrust
SplunkTrust

You should check out the docs around postProcess. In particular if the base search does not contain any transforming search commands splunk will not preserve full information about the events past the 50,000'th event.

Check out the UI Examples app on Splunkbase, and read the page called "Using postProcess on dashboards".

or check out the docs here. http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

The answer in your case is as follows:

instead of having this search

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR")

You want to have this base search:

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") | bin _time span="1min" | stats count by _time responseStatus

Splunk basically will not keep arbitrarily large numbers of events around. On the other hand if the search has transforming commands, it will assemble a complete result set and not cut any corners. Adding these bin and stats commands means that you'll have a much more compact and efficient data set to work with, with no missing information.

View solution in original post

sideview
SplunkTrust
SplunkTrust

You should check out the docs around postProcess. In particular if the base search does not contain any transforming search commands splunk will not preserve full information about the events past the 50,000'th event.

Check out the UI Examples app on Splunkbase, and read the page called "Using postProcess on dashboards".

or check out the docs here. http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

The answer in your case is as follows:

instead of having this search

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR")

You want to have this base search:

apache log search method=get OR method=post | eval responseStatus=case(match(status,"2\d\d"),"OK",match(status,"3\d\d"),"OK",match(status,"4\d\d"),"ERROR",match(status,"5\d\d"),"ERROR") | bin _time span="1min" | stats count by _time responseStatus

Splunk basically will not keep arbitrarily large numbers of events around. On the other hand if the search has transforming commands, it will assemble a complete result set and not cut any corners. Adding these bin and stats commands means that you'll have a much more compact and efficient data set to work with, with no missing information.

sf_user_199
Path Finder

This worked! The only downside to this is I had to do some trickery to get results similar to the top command.

eventstats sum(hit_count) as totalCount | eventstats sum(hit_count) as statusCount by responseStatus | dedup responseStatus | eval percent=(statusCount/totalCount)*100 | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100

0 Karma

bwooden
Splunk Employee
Splunk Employee

I think you'll want to switch your top & your where so that you are getting a top of ERRORs instead of subset of ERRORs from whatever topped.

from

<param name="search">top responseStatus | where match(responseStatus,"ERROR") | gauge percent 0 5 10 100</param>

to

<param name="search">where match(responseStatus,"ERROR") | top responseStatus | gauge percent 0 5 10 100</param>
0 Karma

bwooden
Splunk Employee
Splunk Employee

Yeah, I misread the problem/question

0 Karma

sf_user_199
Path Finder

Unless you think that this is affecting the timechart postprocess, the change you are suggesting doesn't help. I only want the subset of errors.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...