All Apps and Add-ons

Microsoft Azure Add-on: Error when trying to configure users input - is there a limit for importing?

john0499
Explorer

I have tried to configure the AAD Users input, however whenever it runs it fails with the following error:

2020-03-30 23:30:53,493 ERROR pid=10875 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_user.py", line 72, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_user.py", line 29, in collect_events
    users = azutils.get_items(helper, access_token, url)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
RuntimeError: maximum recursion depth exceeded while calling a Python object

We have over 100k users in AAD so I'm wondering if this is more then the add-on can handle?

0 Karma
1 Solution

jbrinkman
Explorer

https://www.geeksforgeeks.org/python-handling-recursion-limit/

The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.

importing the sys module
import sys

the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs

sys.setrecursionlimit(10**6)

*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py

sys.setrecursionlimit(10**6)

*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests

sys.setrecursionlimit(10**6)

View solution in original post

0 Karma

john0499
Explorer

Thanks for the tip! I went to give this a go, but it looks like someone has had a tidy up and we now only have ~80k users in AAD (University - high user turnover). The default settings work fine for 80k, but I'll keep this in mind in case we tip over the limit again.

0 Karma

jbrinkman
Explorer

https://www.geeksforgeeks.org/python-handling-recursion-limit/

The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.

importing the sys module
import sys

the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs

sys.setrecursionlimit(10**6)

*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py

sys.setrecursionlimit(10**6)

*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests

sys.setrecursionlimit(10**6)

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...