All Apps and Add-ons

sourcetype=azure:aad:signin

jaxjohnny2000
Builder

Has anyone see these errors? Do you think these are on the Splunk side or Microsoft side?

2019-11-25 13:57:12,837 ERROR pid=59059 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_signins.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_signins.py", line 77, in collect_events
sign_ins = azutils.get_items(helper, access_token, url)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
HTTPError: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+...

2019-11-25 13:58:17,669 ERROR pid=65098 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_signins.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_signins.py", line 77, in collect_events
sign_ins = azutils.get_items(helper, access_token, url)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
HTTPError: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+...

1 Solution

jaxjohnny2000
Builder

There are two issues in this question. First is the return code 400 and then the return code 429. Both of these are on the Azure side. 429 is by design.

  • NOTE: This response has python code change recommendations. Change at your own risk and peril. 🙂 Please make a backup and know that if you update this app, your changes will be overwritten by that of the author and developer extraordinaire!

Return code 429 is easy, you have exceeded the number of requests per a given time frame due to throttling.

See Misa response in this site: [https://social.msdn.microsoft.com/Forums/azure/en-US/4f6c5141-50ff-4cc0-95c6-05924464859c/microsoft-...]

Status 429 is returned for error "TooManyRequests" that occurs due to throttling. The most common causes of throttling of clients include:

A large number of requests across all applications in a tenant.
A large number of requests from a particular application across all tenants.
Outlook service limits are evaluated for each app ID and mailbox combination. In other words, the limits described apply to a specific app accessing a specific mailbox (user or group). If an application exceeds the limit in one mailbox, it does not affect the ability to access another mailbox.

  • Limit is 10,000 API requests in a 10 minute period for v1.0 and beta endpoints
  • 4 concurrent requests for beta endpoints
  • 15 megabit upload (PATCH, POST, PUT) in a 30 second period for beta endpoints

The next issue, return code 400, is currently a known issue on Microsoft side.

The fix is to modify the code in input_module_MS_AAD_signins.py to change the call to a specific data center. Again, be careful, especially if you do not fully understand python. Do not change any of the indentations in any routine.

“this is known issue as the request is hitting to different region of api while previous request was hitting different region.
The work around is to:
• keep retrying until the request goes to the correct region
• Send all requests to a specific datacenter by including “&gwdc=WCU" as one of the parameters in MsGraph call”

Where WCU is a specific datacenter. See reference below for a list of data center abbreviations.

For example, a call including a specific datacenter would look like this:
https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=DAA&$skiptoken=473ab104ee63d3a2baabcbde7b67e...

Select your data center here:

Reference:
Azure Data centers: https://docs.microsoft.com/en-us/azure/cdn/cdn-pop-abbreviations

While you are in there, you can also change from beta to v1.0

So for Return code 429, this is the before and after somewhere near line 69 and 74 (make a backup copy of the file!!! Test in Dev)

BEFORE:

url = "https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+..." % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ'))

AFTER: changed from beta to v1.0 and inserted the data center

url = "https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=AGA&$orderby=createdDateTime&$filter=created..." % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ'))

View solution in original post

0 Karma

jaxjohnny2000
Builder

There are two issues in this question. First is the return code 400 and then the return code 429. Both of these are on the Azure side. 429 is by design.

  • NOTE: This response has python code change recommendations. Change at your own risk and peril. 🙂 Please make a backup and know that if you update this app, your changes will be overwritten by that of the author and developer extraordinaire!

Return code 429 is easy, you have exceeded the number of requests per a given time frame due to throttling.

See Misa response in this site: [https://social.msdn.microsoft.com/Forums/azure/en-US/4f6c5141-50ff-4cc0-95c6-05924464859c/microsoft-...]

Status 429 is returned for error "TooManyRequests" that occurs due to throttling. The most common causes of throttling of clients include:

A large number of requests across all applications in a tenant.
A large number of requests from a particular application across all tenants.
Outlook service limits are evaluated for each app ID and mailbox combination. In other words, the limits described apply to a specific app accessing a specific mailbox (user or group). If an application exceeds the limit in one mailbox, it does not affect the ability to access another mailbox.

  • Limit is 10,000 API requests in a 10 minute period for v1.0 and beta endpoints
  • 4 concurrent requests for beta endpoints
  • 15 megabit upload (PATCH, POST, PUT) in a 30 second period for beta endpoints

The next issue, return code 400, is currently a known issue on Microsoft side.

The fix is to modify the code in input_module_MS_AAD_signins.py to change the call to a specific data center. Again, be careful, especially if you do not fully understand python. Do not change any of the indentations in any routine.

“this is known issue as the request is hitting to different region of api while previous request was hitting different region.
The work around is to:
• keep retrying until the request goes to the correct region
• Send all requests to a specific datacenter by including “&gwdc=WCU" as one of the parameters in MsGraph call”

Where WCU is a specific datacenter. See reference below for a list of data center abbreviations.

For example, a call including a specific datacenter would look like this:
https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=DAA&$skiptoken=473ab104ee63d3a2baabcbde7b67e...

Select your data center here:

Reference:
Azure Data centers: https://docs.microsoft.com/en-us/azure/cdn/cdn-pop-abbreviations

While you are in there, you can also change from beta to v1.0

So for Return code 429, this is the before and after somewhere near line 69 and 74 (make a backup copy of the file!!! Test in Dev)

BEFORE:

url = "https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+..." % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ'))

AFTER: changed from beta to v1.0 and inserted the data center

url = "https://graph.microsoft.com/v1.0/auditLogs/signIns?gwdc=AGA&$orderby=createdDateTime&$filter=created..." % (query_date, end_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ'))

0 Karma

kjsheltoncmh
New Member

I just started getting these issues on 12/03. Azure SignIns are the only input having the issue. Mine are similar but show a 429 error:

2019-12-06 10:26:31,782 ERROR pid=6452 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_signins.py", line 84, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_signins.py", line 77, in collect_events
    sign_ins = azutils.get_items(helper, access_token, url)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
HTTPError: 429 Client Error:  for url: https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+...
0 Karma

kjsheltoncmh
New Member

Ours randomly started working again. Must've been a Microsoft issue.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...