All Apps and Add-ons

Microsoft Cloud services Addon stops collecting logs

travis_lelle
Explorer

I've experienced the same issue in multiple environments. We're running Splunk Enterprise 6.6.3 and the Microsoft Cloud Services addon. Logs will pull for maybe a day or two, and then we begin to see the following errors in splunk_ta_microsoft-cloudservices_management.log. Typically a reboot will fix the issue, but not all the time.

  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 240, in get_events
    self.do_get_events(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 256, in do_get_events
    events = self.get_one_content(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 154, in get_one_content
    return self._content_request(url=content_info[c.content_uri])
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 124, in _content_request
    raise ome.O365GetContentError(msg + http_resp.msg)
O365GetContentError: Account d3dbea26-263d-4578-bfe4-f300326a3a11_o365 [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] GET request to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d7106
1153612/activity/feed/audit/20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint failed, reason: 403, {"error":{"code":"AF429","message":"Too many requests. Method=GetBlob, Pu
blisherId=00000000-0000-0000-0000-000000000000"}}

2017-11-03 14:59:27,968 +0000 log_level=INFO, pid=29666, tid=Thread-70, file=o365_helper.py, func_name=request, code_line_no=102 | [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] Sending GET request
 to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d71061153612/activity/feed/audit/20171031061205608021143$20171031061205608021143$audit_sharepoint$Audit_SharePoint
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=338 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Start to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=341 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Finish to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=ERROR, pid=29666, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Aud
it.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint"]Failed to get msg from servers=hf1.company.gpsvsoc.com, metric=Audit.SharePoint, error=Traceback (most recent call
 last):
O365GetContentError: [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" p
roxy_enabled="0" ]Fail to get events of content 20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint, stop this round
Tags (1)
1 Solution

Azerty728
Path Finder

It seems a new addon version is on the run on Splunk's side, where it will be possible to change the PublisherID.

Wait & see, stay tuned !

View solution in original post

0 Karma

pedrolito
Explorer

Hi everybody,

I just got the new app from the support team, so ready to test if it's correcting the issue.

Just FYI, the app public release should soon.

Cheers

0 Karma

jaxjohnny2000
Builder

What version did they give you. I downloaded 2.1.0, with the same issues as above.

2018-11-30 18:18:27,066 +0000 log_level=WARNING, pid=42385, tid=Thread-6, file=o365_helper.py, func_name=request, code_line_no=119 | [proxy_type="http" proxy_enabled="0" proxy_rdns="0" ] GET request to https://manage.office.com/api/v1.0/8a807b9b-02da-47f3-a903-791a42a2285c/ServiceComms/CurrentStatus exception, reason Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_helper.py", line 108, in request
body=body, headers=headers)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/init.py", line 1663, in request
(response, content) = self.request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/
init.py", line 1403, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/httplib2/
init_.py", line 1359, in _conn_request
response = conn.getresponse()
File "/opt/splunk/lib/python2.7/httplib.py", line 1121, in getresponse
response.begin()
File "/opt/splunk/lib/python2.7/httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "/opt/splunk/lib/python2.7/httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/opt/splunk/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/opt/splunk/lib/python2.7/ssl.py", line 766, in recv
return self.read(buflen)
File "/opt/splunk/lib/python2.7/ssl.py", line 653, in read
v = self._sslobj.read(len)
SSLError: ('The read operation timed out',)

0 Karma

stonecutter_908
Explorer

I am running the latest beta version of the app given to me by Support and seeing some errors after troubleshooting data that is MIA.

Sample error:
2018-01-11 15:57:16,995 +0000 log_level=ERROR, pid=80730, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="af2a863e-0fb3-462c-80a7-2eddf480771e_Office 365 Management API_Audit.SharePoint" account="TRIM" data="Audit.SharePoint"]Failed to get msg from servers=XXXXXXX, metric=Audit.SharePoint, error=Traceback (most recent call last):
O365GetContentError: [input_name="trim_Office 365 Management API_Audit.SharePoint" account="trim_Splunk_O365_App" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Fail to get events of content 20180107230531796018511$20180107230531796018511$audit_sharepoint$Audit_SharePoint, stop this round

0 Karma

stonecutter_908
Explorer

File itself is Splunk_TA_microsoft-cloudservices-2.0.3.1-3.spl. But it still shows as the GA version of 2.0.3 after install.

0 Karma

pco_support
New Member

Hi stonecutter_908,

Could you give the app version you just tested ?

Thx

Cheers

0 Karma

snrnbrem
Explorer

Hi!
Tagging this as i have the same problem.
Splunk v 6.6.3 and app version 2.0.3.

Azure_AD, Exchange and Sharepoint inputs are getting this error, but the Operational Messages is still functional.

Looking to see if anyone have a good permanent fix to this.

,

0 Karma

pedrolito
Explorer

Hi splunkers,

Same problem as well. Do you know if this problem was also on oldest addon versions ?

Cheers.

0 Karma

Azerty728
Path Finder

It seems a new addon version is on the run on Splunk's side, where it will be possible to change the PublisherID.

Wait & see, stay tuned !

0 Karma

Azerty728
Path Finder

Hi there,

No offical release date. Stay tuned for that. BUT :

I received the addon's patch from Splunk support in order to try (this hotfix is tagged as working by other client, but I didn't get the time to check it on my own).
I encourage you to file a case to Splunk and ask for it if you're impatient enough :D.

I'll update this thread as soon as I've tested it.

Have a nice new year's eve, splunkers 🙂
,Hi there,

No official release date, but you can file a case to Splunk and ask for the hotfix. I received it but didn't have time to test it yet.

I'll update here ASAP if it's working (or not).

Have a nice new year's eve.
Regards

0 Karma

snrnbrem
Explorer

Confirmed that we also got hands on the intermediate version from Splunk. Installed it and it worked like magic!

0 Karma

Azerty728
Path Finder

Good news, it's working !

The patch seems OK : it replaces the PublisherID with our TenantID instead of zeroes, and it works !! I've got plenty of logs now !

I encourage everyone to file a case to Splunk support to ask for the patch, or wait till Splunk releases it officially.

Best regards !

0 Karma

pedrolito
Explorer

Hello Azerty728,

Have you been changing manually the publisherID variable within python code or does the app catch the tenantID and feed get request to MS with it inside ?

I just installed it, and support asked me to put tenantID as publishedID within the code and restart splunk.

I have been waiting for datas but nothing get injected, whereas I can see Get requests done to MS with a publisherID hardcoded into thecode ..

thanks for your help.

Cheers

0 Karma

stonecutter_908
Explorer

Same issue here. Any update on new rev to the Add On?

0 Karma

mgrulke
Explorer

Did they happen to give any possible ETA's on the new add-on update?

0 Karma

Azerty728
Path Finder

They didn't give any info.
I asked them, I filed a case last week about this problem.

I hope we won't be waiting too long.

Cheers

0 Karma

pedrolito
Explorer

Hi Azerty,

Any news about the case you filled ? Or date for beta of new version incoming ?

Thanks a lot

Cheers

0 Karma

travis_lelle
Explorer

Splunk support should be able to provide you with the "experimental" version, which is a fixed version that pretty much takes care of the issue.

0 Karma

mgrulke
Explorer

I ran into the same issue, and I don't see a way to specify the publisher identifier info, we run into throttling errors all the time and pulling down data can be delayed greatly.
See microsofts response below:
The throttling limit are calculated per PublisherIdentifier. If you don’t pass a PublisherIdentifier parameter a Global Identifier 00000000-0000-0000-0000-000000000000 will cause throttling frequently as it will calculate resource usage based on all calls that do not pass a PublisherIdentifier. And this includes calls from other tenants as well.

From the error message I see it looks like you are not passing the PublisherIdentifier parameter so the call uses the global PublisherIdentifier 00000000-0000-0000-0000-000000000000.
To fix this you need to pass a Query string PublisherIdentifier=<> to each and every call of the management API.

0 Karma

Azerty728
Path Finder

Microsoft told me that the PublisherID is a number (preferably the tenant ID). For the PublisherID full of 0, they said that the quota is shared among the same PublisherID connections, and limited to 60k message/minute.
Problem is that I couldn't change this PublisherID in the Splunk Addon to use my tenantID.
The only location I found this suite of zeros among all the addon files was "_serialization.py".
Unfortunately, changing this value to the tenantID and restarting splunk didn't solve the problem.

So if any has another idea...
Microsoft related page containing info about PublisherID :
https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference#api-throttl...

0 Karma

mgrulke
Explorer

Ever happen to find a resolution for this issue? We are having a similar issue trying to connect 365 to splunk error code=AF429 message=Too many requests. Method=GetContents, PublisherId=00000000-0000-0000-0000-000000000000

0 Karma