I've experienced the same issue in multiple environments. We're running Splunk Enterprise 6.6.3 and the Microsoft Cloud Services addon. Logs will pull for maybe a day or two, and then we begin to see the following errors in splunk_ta_microsoft-cloudservices_management.log. Typically a reboot will fix the issue, but not all the time.
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 240, in get_events
self.do_get_events(content_dict)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 256, in do_get_events
events = self.get_one_content(content_dict)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 154, in get_one_content
return self._content_request(url=content_info[c.content_uri])
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 124, in _content_request
raise ome.O365GetContentError(msg + http_resp.msg)
O365GetContentError: Account d3dbea26-263d-4578-bfe4-f300326a3a11_o365 [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] GET request to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d7106
1153612/activity/feed/audit/20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint failed, reason: 403, {"error":{"code":"AF429","message":"Too many requests. Method=GetBlob, Pu
blisherId=00000000-0000-0000-0000-000000000000"}}
2017-11-03 14:59:27,968 +0000 log_level=INFO, pid=29666, tid=Thread-70, file=o365_helper.py, func_name=request, code_line_no=102 | [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] Sending GET request
to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d71061153612/activity/feed/audit/20171031061205608021143$20171031061205608021143$audit_sharepoint$Audit_SharePoint
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=338 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Start to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=341 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Finish to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=ERROR, pid=29666, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Aud
it.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint"]Failed to get msg from servers=hf1.company.gpsvsoc.com, metric=Audit.SharePoint, error=Traceback (most recent call
last):
O365GetContentError: [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" p
roxy_enabled="0" ]Fail to get events of content 20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint, stop this round
It seems a new addon version is on the run on Splunk's side, where it will be possible to change the PublisherID.
Wait & see, stay tuned !
Hi everybody,
I just got the new app from the support team, so ready to test if it's correcting the issue.
Just FYI, the app public release should soon.
Cheers
What version did they give you. I downloaded 2.1.0, with the same issues as above.
2018-11-30 18:18:27,066 +0000 log_level=WARNING, pid=42385, tid=Thread-6, file=o365_helper.py, func_name=request, code_line_no=119 | [proxy_type="http" proxy_enabled="0" proxy_rdns="0" ] GET request to https://manage.office.com/api/v1.0/8a807b9b-02da-47f3-a903-791a42a2285c/ServiceComms/CurrentStatus exception, reason Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_helper.py", line 108, in request
body=body, headers=headers)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/init.py", line 1663, in request
(response, content) = self.request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/init.py", line 1403, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/init_.py", line 1359, in _conn_request
response = conn.getresponse()
File "/opt/splunk/lib/python2.7/httplib.py", line 1121, in getresponse
response.begin()
File "/opt/splunk/lib/python2.7/httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "/opt/splunk/lib/python2.7/httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/opt/splunk/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/opt/splunk/lib/python2.7/ssl.py", line 766, in recv
return self.read(buflen)
File "/opt/splunk/lib/python2.7/ssl.py", line 653, in read
v = self._sslobj.read(len)
SSLError: ('The read operation timed out',)
I am running the latest beta version of the app given to me by Support and seeing some errors after troubleshooting data that is MIA.
Sample error:
2018-01-11 15:57:16,995 +0000 log_level=ERROR, pid=80730, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="af2a863e-0fb3-462c-80a7-2eddf480771e_Office 365 Management API_Audit.SharePoint" account="TRIM" data="Audit.SharePoint"]Failed to get msg from servers=XXXXXXX, metric=Audit.SharePoint, error=Traceback (most recent call last):
O365GetContentError: [input_name="trim_Office 365 Management API_Audit.SharePoint" account="trim_Splunk_O365_App" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Fail to get events of content 20180107230531796018511$20180107230531796018511$audit_sharepoint$Audit_SharePoint, stop this round
File itself is Splunk_TA_microsoft-cloudservices-2.0.3.1-3.spl. But it still shows as the GA version of 2.0.3 after install.
Hi stonecutter_908,
Could you give the app version you just tested ?
Thx
Cheers
Hi!
Tagging this as i have the same problem.
Splunk v 6.6.3 and app version 2.0.3.
Azure_AD, Exchange and Sharepoint inputs are getting this error, but the Operational Messages is still functional.
Looking to see if anyone have a good permanent fix to this.
,
Hi splunkers,
Same problem as well. Do you know if this problem was also on oldest addon versions ?
Cheers.
It seems a new addon version is on the run on Splunk's side, where it will be possible to change the PublisherID.
Wait & see, stay tuned !
Hi there,
No offical release date. Stay tuned for that. BUT :
I received the addon's patch from Splunk support in order to try (this hotfix is tagged as working by other client, but I didn't get the time to check it on my own).
I encourage you to file a case to Splunk and ask for it if you're impatient enough :D.
I'll update this thread as soon as I've tested it.
Have a nice new year's eve, splunkers 🙂
,Hi there,
No official release date, but you can file a case to Splunk and ask for the hotfix. I received it but didn't have time to test it yet.
I'll update here ASAP if it's working (or not).
Have a nice new year's eve.
Regards
Confirmed that we also got hands on the intermediate version from Splunk. Installed it and it worked like magic!
Good news, it's working !
The patch seems OK : it replaces the PublisherID with our TenantID instead of zeroes, and it works !! I've got plenty of logs now !
I encourage everyone to file a case to Splunk support to ask for the patch, or wait till Splunk releases it officially.
Best regards !
Hello Azerty728,
Have you been changing manually the publisherID variable within python code or does the app catch the tenantID and feed get request to MS with it inside ?
I just installed it, and support asked me to put tenantID as publishedID within the code and restart splunk.
I have been waiting for datas but nothing get injected, whereas I can see Get requests done to MS with a publisherID hardcoded into thecode ..
thanks for your help.
Cheers
Same issue here. Any update on new rev to the Add On?
Did they happen to give any possible ETA's on the new add-on update?
They didn't give any info.
I asked them, I filed a case last week about this problem.
I hope we won't be waiting too long.
Cheers
Hi Azerty,
Any news about the case you filled ? Or date for beta of new version incoming ?
Thanks a lot
Cheers
Splunk support should be able to provide you with the "experimental" version, which is a fixed version that pretty much takes care of the issue.
I ran into the same issue, and I don't see a way to specify the publisher identifier info, we run into throttling errors all the time and pulling down data can be delayed greatly.
See microsofts response below:
The throttling limit are calculated per PublisherIdentifier. If you don’t pass a PublisherIdentifier parameter a Global Identifier 00000000-0000-0000-0000-000000000000 will cause throttling frequently as it will calculate resource usage based on all calls that do not pass a PublisherIdentifier. And this includes calls from other tenants as well.
From the error message I see it looks like you are not passing the PublisherIdentifier parameter so the call uses the global PublisherIdentifier 00000000-0000-0000-0000-000000000000.
To fix this you need to pass a Query string PublisherIdentifier=<> to each and every call of the management API.
Microsoft told me that the PublisherID is a number (preferably the tenant ID). For the PublisherID full of 0, they said that the quota is shared among the same PublisherID connections, and limited to 60k message/minute.
Problem is that I couldn't change this PublisherID in the Splunk Addon to use my tenantID.
The only location I found this suite of zeros among all the addon files was "_serialization.py".
Unfortunately, changing this value to the tenantID and restarting splunk didn't solve the problem.
So if any has another idea...
Microsoft related page containing info about PublisherID :
https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference#api-throttl...
Ever happen to find a resolution for this issue? We are having a similar issue trying to connect 365 to splunk error code=AF429 message=Too many requests. Method=GetContents, PublisherId=00000000-0000-0000-0000-000000000000