topic Why is my real-time post process dashboard showing different results from running the same search directly? in Splunk Search

Why is my real-time post process dashboard showing different results from running the same search directly?

DennisMohn — Mon, 07 Sep 2015 14:14:46 GMT

Hi folks,

I'm experiencing a strange behavior on one of my splunk real-time postprocess dashboards. The numbers shown are significantly smaller as when I run the same search directly.

Code for the dashboard:

<search id="allcount">
    <query>sourcetype=mgw_live | fields host,receiver,http_status</query>
    <earliest>rt-60m</earliest>
    <latest>rt</latest>
 </search>

<single>
   <title>PRD2</title>
   <search base="allcount">
     <query>search host=prd2 | stats count</query>
   </search>
   <option name="underLabel">Datagramme</option>
   <option name="field">count</option>
   <option name="linkView">search</option>
   <option name="drilldown">none</option>
</single>

Dashboard is showing a count of about 8000 to 9000 events.

If I run the same search directly

sourcetype=mgw_live | fields host,receiver,http_status | search host=prd2 | stats count

I'm getting about 67.500 results which is much more likely, if I compare it to the source file.

What could be the reason for this?

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

HiroshiSatoh — Tue, 08 Sep 2015 00:39:38 GMT

Did you look at to check the conditions in 「Search job inspector」?

Is there a difference in the 「Search job properties」？
ex.earliestTime　OR　latestTime

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

DennisMohn — Wed, 09 Sep 2015 08:11:46 GMT

Dashboard job:
earliestTime: 2015-09-09T09:06:46.000+02:00
latestTime: 2015-09-09T10:06:46.000+02:00

Free Search Job:
earliestTime: 2015-09-09T09:08:24.000+02:00
latestTime: 2015-09-09T10:08:24.000+02:00

I also recognize, that the Event counts in the Dashboard Job differ from the displayes results:

eventAvailableCount 71699
eventCount 71699

I assume the error is within the postprocessing command. Is there any chance to inspect, what the postprocess does?

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

DennisMohn — Wed, 09 Sep 2015 13:01:16 GMT

I have re-evaluated the issue. If the timeframe is very short (earliest=rt-5m, latest=rtnow) the results are the same. As soon as I increase the searchtime, the results start to vary.

Interval 5min => 1072 results (dashboard) vs. 1073 results (search) => both real-time changing, OK!
Interval 10min => 1850 results vs. 2280 results
Interval 30min => 1672 results vs. 6251 results
Interval 60min => 1875 results va. 12046

I don't see any reason why the real-time dashboard starts to drop results if the interval increases...

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

HiroshiSatoh — Thu, 10 Sep 2015 03:36:37 GMT

Log There is also a link to "search.log" to the top of the inspector.

Has been output is the number of the search process on the information in the "job inspector".
Please see what the difference in the number has come out at any stage.

http://docs.splunk.com/Documentation/Splunk/6.2.5/Knowledge/ViewsearchjobpropertieswiththeJobInspector

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

HiroshiSatoh — Fri, 11 Sep 2015 00:07:29 GMT

For example, What happens if you specify the index results?

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

DennisMohn — Fri, 11 Sep 2015 13:11:41 GMT

I don't see any errors in the search.log

What do you mean by "specifying index results"?

Re: Why is my real-time post process dashboard showing different results from running the same search directly?

HiroshiSatoh — Thu, 24 Sep 2015 10:05:09 GMT

Any errors found?

Make sure there is no difference in the index that are used in the search.
There may be a difference in the index to use in the difference of authority.