I have tried to configure the AAD Users input, however whenever it runs it fails with the following error:
2020-03-30 23:30:53,493 ERROR pid=10875 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_user.py", line 72, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_user.py", line 29, in collect_events
users = azutils.get_items(helper, access_token, url)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
RuntimeError: maximum recursion depth exceeded while calling a Python object
We have over 100k users in AAD so I'm wondering if this is more then the add-on can handle?
https://www.geeksforgeeks.org/python-handling-recursion-limit/
The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.
importing the sys module
import sys
the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs
sys.setrecursionlimit(10**6)
*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py
sys.setrecursionlimit(10**6)
*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests
sys.setrecursionlimit(10**6)
Thanks for the tip! I went to give this a go, but it looks like someone has had a tidy up and we now only have ~80k users in AAD (University - high user turnover). The default settings work fine for 80k, but I'll keep this in mind in case we tip over the limit again.
https://www.geeksforgeeks.org/python-handling-recursion-limit/
The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.
importing the sys module
import sys
the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs
sys.setrecursionlimit(10**6)
*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py
sys.setrecursionlimit(10**6)
*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests
sys.setrecursionlimit(10**6)