Microsoft Azure Add-on: Error when trying to configure users input - is there a limit for importing?

[john0499]

I have tried to configure the AAD Users input, however whenever it runs it fails with the following error:

2020-03-30 23:30:53,493 ERROR pid=10875 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_user.py", line 72, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_user.py", line 29, in collect_events
    users = azutils.get_items(helper, access_token, url)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
RuntimeError: maximum recursion depth exceeded while calling a Python object

We have over 100k users in AAD so I'm wondering if this is more then the add-on can handle?

[john0499]

Thanks for the tip! I went to give this a go, but it looks like someone has had a tidy up and we now only have ~80k users in AAD (University - high user turnover). The default settings work fine for 80k, but I'll keep this in mind in case we tip over the limit again.

[jbrinkman · Accepted solution]

https://www.geeksforgeeks.org/python-handling-recursion-limit/

The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.

importing the sys module
import sys

the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs

sys.setrecursionlimit(10**6)

*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py

sys.setrecursionlimit(10**6)

*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests

sys.setrecursionlimit(10**6)