All Apps and Add-ons

Microsoft Azure Add-on: Error when trying to configure users input - is there a limit for importing?

john0499
Explorer

I have tried to configure the AAD Users input, however whenever it runs it fails with the following error:

2020-03-30 23:30:53,493 ERROR pid=10875 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_user.py", line 72, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_user.py", line 29, in collect_events
    users = azutils.get_items(helper, access_token, url)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
RuntimeError: maximum recursion depth exceeded while calling a Python object

We have over 100k users in AAD so I'm wondering if this is more then the add-on can handle?

0 Karma
1 Solution

jbrinkman
Explorer

https://www.geeksforgeeks.org/python-handling-recursion-limit/

The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.

importing the sys module
import sys

the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs

sys.setrecursionlimit(10**6)

*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py

sys.setrecursionlimit(10**6)

*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests

sys.setrecursionlimit(10**6)

View solution in original post

0 Karma

john0499
Explorer

Thanks for the tip! I went to give this a go, but it looks like someone has had a tidy up and we now only have ~80k users in AAD (University - high user turnover). The default settings work fine for 80k, but I'll keep this in mind in case we tip over the limit again.

0 Karma

jbrinkman
Explorer

https://www.geeksforgeeks.org/python-handling-recursion-limit/

The “sys” module in Python provides a function called setrecursionlimit() to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually 10^4. If you are dealing with large inputs, you can set it to, 10^6 so that large inputs can be handled without any errors.

importing the sys module
import sys

the setrecursionlimit function is used to modify the default recursion limit set by python. Using this, we can increase the recursion limit to satisfy our needs

sys.setrecursionlimit(10**6)

*** To get our azure:aad:user data to ingest we added the following change to $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py

sys.setrecursionlimit(10**6)

*** We placed it following all of the imports
import sys
import json
import datetime
import dateutil.parser
import urlparse
import requests

sys.setrecursionlimit(10**6)

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...