Microsoft o365 error: The read operation timed out

[dyeyniyel]

Hello, 

We get this error and I'm not entirely sure on how we can resolve this? Looks like a timeout issue

2021-09-01 14:07:44,983 level=ERROR pid=12315 tid=Thread-20 logger=splunk_ta_o365.modinputs.management_activity pos=management_activity.py:do:159 | datainput=b'at_rbi_management_activity_sharepoint' start_time=1630505194 | message="Failed to retrieve content blob." content_id=b'20210901140215154005183$20210901140215154005183$audit_sharepoint$Audit_SharePoint$emea0023'
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 383, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/splunk/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/opt/splunk/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/opt/splunk/lib/python3.7/http/client.py", line 271, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/splunk/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/splunk/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 637, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 599, in urlopen
    chunked=chunked)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 385, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/urllib3/connectionpool.py", line 305, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='manage.office.com', port=443): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 153, in do
    response = self._subscription.retrieve_content_blob(session, content.uri)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 166, in retrieve_content_blob
    return self._request(session, 'GET', url)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 179, in _request
    response = session.request(method, url, params=params, timeout=self._request_timeout)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='manage.office.com', port=443): Read timed out. (read timeout=60)

[shivanshu1593 · Accepted solution]

Hello @dyeyniyel ,

While calling the Office 365 management API, the modular input at times needs more than the default 60 seconds to perform the transactions or sometimes gets hanged during data collection.

To remedy it, you'll have to tune the request_timeout parameter for the Add On. Please follow the below mentioned steps:

1. Navigate to $SPLUNK_HOME/etc/apps/splunk_ta_o365/local, and create an inputs.conf file, if it does not already exist.

2. Copy below stanza, and add it to the $SPLUNK_HOME/etc/apps/splunk_ta_o365/local/inputs.conf file.

[splunk_ta_o365_management_activity]

request_timeout = 300 

3. Save the file and restart splunkd. Please wait for the input to run again and see if you still end up encountering this error.

Let me know if this helps.

Thanks,

S

***If this helped, please accept it as a solution. It helps others to find the solution for similar issues quickly.***

[sahilsingh]

Hi Shiv, I tried to update the request_timeout = 300 but still getting the similar error.