Report time offsets are off by 5 minutes?

[doctorjw]

I have a report scheduled to run every 5 minutes (*/5 .....). This report gathers summary data from 6 minutes ago, to 1 minute ago, like this (I've removed the index & search criteria, etc., as they aren't germane):

index=<index> <search_criteria> earliest=-6m@m latest=-1m@m ....

When I run this with a "collect' with testmode=true, I get exactly what I'd expect ... If the current time is 18:00, I get data for 17:54, 17:55, 17:56, 17:57, and 17:58

The same query run as a report, though, with testmode=false, apparently the offset times are changed by 5 minutes and I get data for 17:49, 17:50, 17:51, 17:52, and 17:53

Is there something in the configuration that would have report offsets be altered by 5 minutes? Something I'm missing dealing with reports somehow? Running interactively vs running in a report is clearly offsetting things by 5 minutes.

I've used this query to verify the times in my report query:

index=<summary index> source="summary_revenue_by_minute"
| convert ctime(info_min_time) AS minTime
| convert ctime(info_max_time) AS maxTime
| convert ctime(info_search_time) AS searchTime
| sort by -_time
| table _time minTime maxTime searchTime _raw

I'm scratching my head & reading docs to no avail. Thoughts?

[doctorjw]

Could the answer be that the saved search has an "allow_skew" of 5m?

8^/

dr_j

[doctorjw]

This does look like the issue; we have a default allow_skew of 5m for some reason. Testing with an allow_skew of 0 fixes this, and ... my data lines up with what I expected to have.

[doctorjw]

Well ... I take this back. When I do a manual edit of my scheduled job and set allow_skew to 0, I get what I expect. However, when our pipeline deploys a report (so I can't edit it directly) but my yaml file has allow_skew 0 .. I'm not getting the changes I expect.

I feel I'm closer but ... not there yet.

[doctorjw]

Okay, yes, this is solved.

Using a yaml file to put the report in place, our default value of 5m for the allow_skew comes into play, regardless of what I set in my yaml file. I haven't found a way to reset that value this way; I can manually edit a report outside of our report area to set the allow_skew, though.

To get things to work, I just need to be much more particular about the cron, and use a cron expression outside of what allow_skew will adjust ... and that's fine.

[Nisha18789]

hi @doctorjw , could you check when the scheduled report is running and the scheduled time , using below query?

index=_internal sourcetype=scheduler savedsearch_name=<your report name> status=success

[doctorjw]

Okay, running this query:

index=_internal sourcetype=scheduler savedsearch_name=dip-summary-revenue-by-minute status=success

I get this result:

9/1/20 9:24:24.796 AM

09-01-2020 14:24:24.796 +0000 INFO SavedSplunker - savedsearch_id="nobody;dip-summary-revenue-by-minute", search_type="scheduled", user="admin", app="myapp", savedsearch_name="dip-summary-revenue-by-minute", priority=default, status=success, digest_mode=1, scheduled_time=1598970000, window_time=-1, dispatch_time=1598970264, run_time=0.497, result_count=5, alert_actions="", sid="scheduler__admin__myapp__RMD5a5372a05036e6d57_at_1598970000_39293", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

When I run my query to get the latest data right after running the above query:

index=prod-summary-revenue source="summary_revenue_by_minute"
| convert ctime(info_min_time) AS minTime
| convert ctime(info_max_time) AS maxTime
| convert ctime(info_search_time) AS searchTime
| sort by -_time
| table _time minTime maxTime searchTime _raw

I get these results:

2020-09-01 09:18:00	09/01/2020 09:14:00.000	09/01/2020 09:19:00.000	09/01/2020 09:24:24.104	09/01/2020 09:18:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=171, revenueByMinute="909.96"
2020-09-01 09:17:00	09/01/2020 09:14:00.000	09/01/2020 09:19:00.000	09/01/2020 09:24:24.104	09/01/2020 09:17:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=167, revenueByMinute="882.49"
2020-09-01 09:16:00	09/01/2020 09:14:00.000	09/01/2020 09:19:00.000	09/01/2020 09:24:24.104	09/01/2020 09:16:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=150, revenueByMinute="830.47"
2020-09-01 09:15:00	09/01/2020 09:14:00.000	09/01/2020 09:19:00.000	09/01/2020 09:24:24.104	09/01/2020 09:15:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=122, revenueByMinute="606.37"

Confirming that instead of the generated data being in the 1-6 minute old range, it's 6-11 minutes old (offset 5 minutes in the past from what we expected).

btw, not sure it's that important to mention, but this is with splunk cloud.

Dr_j

[Nisha18789]

@doctorjw , looking at the difference in search time and event time, it definately looks like a case of delayed report by scheduler. When you run manually , its a adhoc search having the highest priority and hence it works fine. While when scheduler does the execution , it considers a number of parameters if there are a large number of searches needs to be run at the same time, one of the paramter is search priority which I see it set as default for this report, could you please try to increase the priority of this search as highest and check if that fixes the issue. You can also try updating the Schedule Window which is also one of paramter , scheduler considers.

09-01-2020 14:24:24.796 +0000 INFO SavedSplunker - savedsearch_id="nobody;dip-summary-revenue-by-minute", search_type="scheduled", user="admin", app="myapp", savedsearch_name="dip-summary-revenue-by-minute", priority=default,