I'm experiencing a strange behavior on one of my splunk real-time postprocess dashboards. The numbers shown are significantly smaller as when I run the same search directly.
Code for the dashboard:
<search id="allcount"> <query>sourcetype=mgw_live | fields host,receiver,http_status</query> <earliest>rt-60m</earliest> <latest>rt</latest> </search> <single> <title>PRD2</title> <search base="allcount"> <query>search host=prd2 | stats count</query> </search> <option name="underLabel">Datagramme</option> <option name="field">count</option> <option name="linkView">search</option> <option name="drilldown">none</option> </single>
Dashboard is showing a count of about 8000 to 9000 events.
If I run the same search directly
sourcetype=mgw_live | fields host,receiver,http_status | search host=prd2 | stats count
I'm getting about 67.500 results which is much more likely, if I compare it to the source file.
What could be the reason for this?
I have re-evaluated the issue. If the timeframe is very short (earliest=rt-5m, latest=rtnow) the results are the same. As soon as I increase the searchtime, the results start to vary.
Interval 5min => 1072 results (dashboard) vs. 1073 results (search) => both real-time changing, OK!
Interval 10min => 1850 results vs. 2280 results
Interval 30min => 1672 results vs. 6251 results
Interval 60min => 1875 results va. 12046
I don't see any reason why the real-time dashboard starts to drop results if the interval increases...
Free Search Job:
I also recognize, that the Event counts in the Dashboard Job differ from the displayes results:
I assume the error is within the postprocessing command. Is there any chance to inspect, what the postprocess does?
Log There is also a link to "search.log" to the top of the inspector.
Has been output is the number of the search process on the information in the "job inspector".
Please see what the difference in the number has come out at any stage.