Try this from the management console so it runs against all the servers.    | rest /services/apps/local search="disabled=0" count=0 splunk_server=* | table title core | search core=1 ... View more

We found the answer. Don't go big, go small! Because there was so much data coming in, a larger batch size simply buffered in memory, and then crashed. So, decreasing the batch size allows the add-on to finish, and then go ahead and write to disk. We also brought the polling interval down, so once it completes one batch, it will almost immediately start a new one. We also increased the partitions (which means a new event hub) to 20, and the thread count to 20. So The add-on will have plenty of threads to bring in a lot of small batches, very quickly. so, the moral of the story? Bigger is not always better. ... View more

There are two issues in this question. First is the return code 400 and then the return code 429. Both of these are on the Azure side. 429 is by design. NOTE: This response has python code change recommendations. Change at your own risk and peril. 🙂 Please make a backup and know that if you update this app, your changes will be overwritten by that of the author and developer extraordinaire! Return code 429 is easy, you have exceeded the number of requests per a given time frame due to throttling. See Misa response in this site: [https://social.msdn.microsoft.com/Forums/azure/en-US/4f6c5141-50ff-4cc0-95c6-05924464859c/microsoft-graph-api-keeps-returning-429][1] Status 429 is returned for error "TooManyRequests" that occurs due to throttling. The most common causes of throttling of clients include: A large number of requests across all applications in a tenant. A large number of requests from a particular application across all tenants. Outlook service limits are evaluated for each app ID and mailbox combination. In other words, the limits described apply to a specific app accessing a specific mailbox (user or group). If an application exceeds the limit in one mailbox, it does not affect the ability to access another mailbox. Limit is 10,000 API requests in a 10 minute period for v1.0 and beta endpoints 4 concurrent requests for beta endpoints 15 megabit upload (PATCH, POST, PUT) in a 30 second period for beta endpoints The next issue, return code 400, is currently a known issue on Microsoft side. The fix is to modify the code in input_module_MS_AAD_signins.py to change the call to a specific data center. Again, be careful, especially if you do not fully understand python. Do not change any of the indentations in any routine. “this is known issue as the request is hitting to different region of api while previous request was hitting different region. The work around is to: • keep retrying until the request goes to the correct region • Send all requests to a specific datacenter by including “&gwdc=WCU" as one of the parameters in MsGraph call” Where WCU is a specific datacenter. See reference below for a list of data center abbreviations. For example, a call including a specific datacenter would look like this: https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=DAA&$skiptoken=473ab104ee63d3a2baabcbde7b67e94c04_1000 Select your data center here: Reference: Azure Data centers: https://docs.microsoft.com/en-us/azure/cdn/cdn-pop-abbreviations While you are in there, you can also change from beta to v1.0 So for Return code 429, this is the before and after somewhere near line 69 and 74 (make a backup copy of the file!!! Test in Dev) BEFORE: url = "https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+ge+%s+and+createdDateTime+le+%s" % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ')) AFTER: changed from beta to v1.0 and inserted the data center url = "https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=AGA&$orderby=createdDateTime&$filter=createdDateTime+ge+%s+and+createdDateTime+le+%s" % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ')) ... View more