Microsoft Office 365 Reporting Web Service ends with "_Splunk_ max date before getting message"

[ldnail_at_TI]

Microsoft Office 365 Reporting Web Service works fine with an "Index Once" config where Start date/time & End date/time are defined.

Set this to Continuously Monitor and it appears to fail... 

This connector is defaults with empty start or end date/time fields

2022-10-21 13:36:54,969 INFO pid=15262 tid=MainThread file=splunk_rest_client.py:_request_handler:99 | Use HTTP connection pooling
2022-10-21 13:36:54,970 DEBUG pid=15262 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2022-10-21 13:36:54,971 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_new_conn:941 | Starting new HTTPS connection (1): 127.0.0.1:8089
2022-10-21 13:36:54,973 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_make_request:442 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5564
2022-10-21 13:36:54,974 DEBUG pid=15262 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'offset': 0, 'search': 'TA_MS_O365_Reporting_checkpointer'})
2022-10-21 13:36:54,974 DEBUG pid=15262 tid=MainThread file=binding.py:new_f:74 | Operation took 0:00:00.003694
2022-10-21 13:36:54,976 DEBUG pid=15262 tid=MainThread file=binding.py:new_f:74 | Operation took 0:00:00.002273
2022-10-21 13:36:54,976 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_make_request:442 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&offset=0&search=TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 4716
2022-10-21 13:36:54,978 DEBUG pid=15262 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/O365_Message_Trace_obj_checkpoint_oauth (body: {})
2022-10-21 13:36:54,979 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_make_request:442 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/O365_Message_Trace_obj_checkpoint_oauth HTTP/1.1" 404 140
2022-10-21 13:36:54,980 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_new_conn:941 | Starting new HTTPS connection (1): login.windows.net:443
2022-10-21 13:36:54,980 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Proxy is enabled: web:8080
2022-10-21 13:36:54,980 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Getting proxy server.
2022-10-21 13:36:54,980 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ message trace URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2022-10-16T13:36:54.979985Z' and EndDate eq datetime'2022-10-16T14:36:54.979985Z'
2022-10-21 13:36:54,980 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Start date: 2022-10-16 13:36:54.979985, End date: 2022-10-16 14:36:54.979985
2022-10-21 13:36:55,142 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_make_request:442 | https://login.windows.net:443 "POST /2445612c-659f-4f0e-a8b2-51087c624102/oauth2/token HTTP/1.1" 200 1815
2022-10-21 13:36:55,144 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Proxy is enabled: web:8080
2022-10-21 13:36:55,144 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Getting proxy server.
2022-10-21 13:36:55,145 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_new_conn:941 | Starting new HTTPS connection (1): reports.office365.com:443
2022-10-21 13:36:59,928 DEBUG pid=15262 tid=MainThread file=connectionpool.py:_make_request:442 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2022-10-16T13:36:54.979985Z'%20and%20EndDate%20eq%20datetime'2022-10-16T14:36:54.979985Z' HTTP/1.1" 200 216
2022-10-21 13:36:59,930 DEBUG pid=15262 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ max date before getting message: 2022-10-16 13:36:54.979985

I changed the Start date/time 2022-10-19 00:00:00 2 full days ago, so I don't bump against the 7 day boundary.

2022-10-21 13:40:31,102 DEBUG pid=15810 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ message trace URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2022-10-19T00:00:00Z' and EndDate eq datetime'2022-10-19T01:00:00Z'
2022-10-21 13:40:31,102 DEBUG pid=15810 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Start date: 2022-10-19 00:00:00, End date: 2022-10-19 01:00:00
2022-10-21 13:40:31,103 DEBUG pid=15810 tid=MainThread file=connectionpool.py:_new_conn:941 | Starting new HTTPS connection (1): login.windows.net:443
2022-10-21 13:40:31,339 DEBUG pid=15810 tid=MainThread file=connectionpool.py:_make_request:442 | https://login.windows.net:443 "POST /2445612c-659f-4f0e-a8b2-51087c624102/oauth2/token HTTP/1.1" 200 1815
2022-10-21 13:40:31,341 DEBUG pid=15810 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Proxy is enabled: web:8080
2022-10-21 13:40:31,341 DEBUG pid=15810 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ Getting proxy server.
2022-10-21 13:40:31,342 DEBUG pid=15810 tid=MainThread file=connectionpool.py:_new_conn:941 | Starting new HTTPS connection (1): reports.office365.com:443
2022-10-21 13:40:34,302 DEBUG pid=15810 tid=MainThread file=connectionpool.py:_make_request:442 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2022-10-19T00:00:00Z'%20and%20EndDate%20eq%20datetime'2022-10-19T01:00:00Z' HTTP/1.1" 200 122
2022-10-21 13:40:34,303 DEBUG pid=15810 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ max date before getting message: 2022-10-19 00:00:00

I've not been able to determine what the comment "_Splunk_ max date before getting message: <2022-10-19 00:00:00>"

The lookup TA_MS_O365_Reporting_checkpointer shows a row with _key <nameofinput>_once_checkpoint_oauth which looks to be from when I did the Index Once.

Would some who's running Continuously Monitor please take a look into lookup TA_MS_O365_Reporting_checkpointer & let me know what _key name & state columns indicate for where _key = *_checkpoint_*

Of course, if someone has experienced the same & figured this out, I'd appreciate any words of wisdom.