Splunk Search

Why is my real-time post process dashboard showing different results from running the same search directly?

DennisMohn
Path Finder

Hi folks,

I'm experiencing a strange behavior on one of my splunk real-time postprocess dashboards. The numbers shown are significantly smaller as when I run the same search directly.

Code for the dashboard:

<search id="allcount">
    <query>sourcetype=mgw_live | fields host,receiver,http_status</query>
    <earliest>rt-60m</earliest>
    <latest>rt</latest>
 </search>

<single>
   <title>PRD2</title>
   <search base="allcount">
     <query>search host=prd2 | stats count</query>
   </search>
   <option name="underLabel">Datagramme</option>
   <option name="field">count</option>
   <option name="linkView">search</option>
   <option name="drilldown">none</option>
</single>

Dashboard is showing a count of about 8000 to 9000 events.

If I run the same search directly

sourcetype=mgw_live | fields host,receiver,http_status | search host=prd2 | stats count

I'm getting about 67.500 results which is much more likely, if I compare it to the source file.

What could be the reason for this?

DennisMohn
Path Finder

I have re-evaluated the issue. If the timeframe is very short (earliest=rt-5m, latest=rtnow) the results are the same. As soon as I increase the searchtime, the results start to vary.

Interval 5min => 1072 results (dashboard) vs. 1073 results (search) => both real-time changing, OK!
Interval 10min => 1850 results vs. 2280 results
Interval 30min => 1672 results vs. 6251 results
Interval 60min => 1875 results va. 12046

I don't see any reason why the real-time dashboard starts to drop results if the interval increases...

0 Karma

HiroshiSatoh
Champion

Did you look at to check the conditions in 「Search job inspector」?

alt text

Is there a difference in the 「Search job properties」?
ex.earliestTime OR latestTime

DennisMohn
Path Finder

Dashboard job:
earliestTime: 2015-09-09T09:06:46.000+02:00
latestTime: 2015-09-09T10:06:46.000+02:00

Free Search Job:
earliestTime: 2015-09-09T09:08:24.000+02:00
latestTime: 2015-09-09T10:08:24.000+02:00

I also recognize, that the Event counts in the Dashboard Job differ from the displayes results:

eventAvailableCount 71699
eventCount 71699

I assume the error is within the postprocessing command. Is there any chance to inspect, what the postprocess does?

0 Karma

HiroshiSatoh
Champion

Log There is also a link to "search.log" to the top of the inspector.

Has been output is the number of the search process on the information in the "job inspector".
Please see what the difference in the number has come out at any stage.

http://docs.splunk.com/Documentation/Splunk/6.2.5/Knowledge/ViewsearchjobpropertieswiththeJobInspect...

0 Karma

HiroshiSatoh
Champion

For example, What happens if you specify the index results?

0 Karma

DennisMohn
Path Finder

I don't see any errors in the search.log

What do you mean by "specifying index results"?

0 Karma

HiroshiSatoh
Champion

Any errors found?

Make sure there is no difference in the index that are used in the search.
There may be a difference in the index to use in the difference of authority.

0 Karma