Why is output on a Dashboard Panel is different from Normal search using the same query?

[bhaskargsky]

PS: I checked the other similar questions in this forum but my question is not answered. Hence posting again.

Good afternoon all,

When I try to fetch my events using a panel in a dashboard, the results do come up but they are way small compared to the exact same query that I use in search window. The purpose is to keep monitoring failure spikes happening on the production environment.

I'm providing the normal search query as well as the entire dashboard query (scrubbed) for your review. For sure the normal search is returning more values than the exact query as run from Panel.

Normal search query: 

index=myOnlyIndex
| fields * | search clientApp="Fictitious Company Mobile App*" Customer=Production app IN (https) requestMethod=POST
| | timechart span=1h count BY failureCode usenull=false useother=false

Dashboard details:

<form version="1.1" theme="light">

<label>Production Spikes.</label>

<description>This is dashboard contains various panels that will be tested before eventually moved into functional dashboard</description>

<search id="BaseSearch">

<query>index=myOnlyIndex
| fields *
</query>

<earliest>$time_token.earliest$</earliest>

<latest>$time_token.latest$</latest>

</search>

<fieldset submitButton="true" autoRun="false">

<input type="time" token="time_token" searchWhenChanged="true">
<label>Timeframe</label>
<default>
<earliest>@d</earliest>
<latest>now</latest>
</default>
</input>

<input type="dropdown" token="environment" searchWhenChanged="true">
<label>Environment</label>
<prefix>Customer=</prefix>
<suffix></suffix>
<initialValue>*</initialValue>
<choice value="*">All</choice>
<choice value="UAT">UAT</choice>
<choice value="Production">Production</choice>
</input>

<input type="dropdown" token="protocol" searchWhenChanged="true">
<label>Protocol</label>
<prefix>app IN (</prefix>
<suffix>)</suffix>
<choice value="http, https">Both</choice>
<choice value="https">HTTPS Only</choice>
<choice value="http">HTTP Only</choice>
</input>

<input type="dropdown" token="reqMethod" searchWhenChanged="true">
<label>Request Method</label>
<prefix>requestMethod=</prefix>
<suffix></suffix>
<initialValue>*</initialValue>
<choice value="*">All</choice>
<choice value="GET">GET Method</choice>
<choice value="POST">POST Method</choice>
<choice value="PUT">PUT Method</choice>
</input>

</fieldset>

<row>

<panel>

<title>Mobile Application Generic Spikes</title>
<chart>
<search base="BaseSearch">
<query>search clientApp="Fictitious Company Mobile App*" $environment$ $protocol$ $reqMethod$
| timechart span=1h count BY failureCode usenull=false useother=false
</query>
</search>
</chart>

</panel>

</row>

</form>

[bowesmana]

It looks like you are using a base search that doesn't really do anything. Your base search that just does

<search id="BaseSearch">
  <query>index=myOnlyIndex
| fields *
  </query>
  <earliest>$time_token.earliest$</earliest>
  <latest>$time_token.latest$</latest>
</search>

is not what base searches are intended for - this can actually make performance worse. Base searches will be limited to 500,000 events, so as you are doing no aggregations in your base search, then your post process search will only process 500,000 events and ignore the rest - see

https://docs.splunk.com/Documentation/Splunk/9.1.0/Viz/Savedsearches#Use_a_transforming_base_search

which describes this behaviour. 

[isoutamo]

Hi

this is exactly like @bowesmana said. You should have transforming search in you base search. 

I think that you could fix this by something like this 

<search id="BaseSearch">
 <query>index=myOnlyIndex
   | bin span=1h _time
   | stats count by _time failureCode clientApp environment protocol reqMethods
 </query>
 <earliest>$time_token.earliest$</earliest>
 <latest>$time_token.latest$</latest>
</search>

Then I think that your post-process search could work like it's now.

You just need to ensure that base search don't return more than 500k events and it's don't take longer than 60s.

r. Ismo