topic Re: Microsoft Cloud services Addon stops collecting logs in All Apps and Add-ons

Microsoft Cloud services Addon stops collecting logs

travis_lelle — Tue, 29 Sep 2020 16:33:52 GMT

I've experienced the same issue in multiple environments. We're running Splunk Enterprise 6.6.3 and the Microsoft Cloud Services addon. Logs will pull for maybe a day or two, and then we begin to see the following errors in splunk_ta_microsoft-cloudservices_management.log. Typically a reboot will fix the issue, but not all the time.

  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 240, in get_events
    self.do_get_events(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 256, in do_get_events
    events = self.get_one_content(content_dict)
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 154, in get_one_content
    return self._content_request(url=content_info[c.content_uri])
  File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/o365_content.py", line 124, in _content_request
    raise ome.O365GetContentError(msg + http_resp.msg)
O365GetContentError: Account d3dbea26-263d-4578-bfe4-f300326a3a11_o365 [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] GET request to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d7106
1153612/activity/feed/audit/20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint failed, reason: 403, {"error":{"code":"AF429","message":"Too many requests. Method=GetBlob, Pu
blisherId=00000000-0000-0000-0000-000000000000"}}

2017-11-03 14:59:27,968 +0000 log_level=INFO, pid=29666, tid=Thread-70, file=o365_helper.py, func_name=request, code_line_no=102 | [proxy_type="http" proxy_rdns="0" proxy_enabled="0" ] Sending GET request
 to https://manage.office.com/api/v1.0/cc03cb3f-e51d-4fb2-b5f4-d71061153612/activity/feed/audit/20171031061205608021143$20171031061205608021143$audit_sharepoint$Audit_SharePoint
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=338 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Start to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=INFO, pid=29666, tid=Thread-6, file=o365_content.py, func_name=tear_down, code_line_no=341 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint
" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Finish to tear down, wait=False
2017-11-03 14:59:27,991 +0000 log_level=ERROR, pid=29666, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Aud
it.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint"]Failed to get msg from servers=hf1.company.gpsvsoc.com, metric=Audit.SharePoint, error=Traceback (most recent call
 last):
O365GetContentError: [input_name="d3dbea26-263d-4578-bfe4-f300326a3a11_o365_Audit.SharePoint" account="d3dbea26-263d-4578-bfe4-f300326a3a11_o365" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" p
roxy_enabled="0" ]Fail to get events of content 20171031061141455019716$20171031061141455019716$audit_sharepoint$Audit_SharePoint, stop this round

Re: Microsoft Cloud services Addon stops collecting logs

mgrulke — Tue, 14 Nov 2017 15:57:53 GMT

Ever happen to find a resolution for this issue? We are having a similar issue trying to connect 365 to splunk error code=AF429 message=Too many requests. Method=GetContents, PublisherId=00000000-0000-0000-0000-000000000000

Re: Microsoft Cloud services Addon stops collecting logs

travis_lelle — Tue, 14 Nov 2017 15:59:20 GMT

Nothing. I have a support ticket open with Splunk.

Re: Microsoft Cloud services Addon stops collecting logs

Azerty728 — Thu, 16 Nov 2017 08:09:52 GMT

Microsoft told me that the PublisherID is a number (preferably the tenant ID). For the PublisherID full of 0, they said that the quota is shared among the same PublisherID connections, and limited to 60k message/minute.
Problem is that I couldn't change this PublisherID in the Splunk Addon to use my tenantID.
The only location I found this suite of zeros among all the addon files was "_serialization.py".
Unfortunately, changing this value to the tenantID and restarting splunk didn't solve the problem.

So if any has another idea...
Microsoft related page containing info about PublisherID :
https://msdn.microsoft.com/en-us/office-365/office-365-management-activity-api-reference#api-throttling

Re: Microsoft Cloud services Addon stops collecting logs

mgrulke — Tue, 21 Nov 2017 22:15:43 GMT

I ran into the same issue, and I don't see a way to specify the publisher identifier info, we run into throttling errors all the time and pulling down data can be delayed greatly.
See microsofts response below:
The throttling limit are calculated per PublisherIdentifier. If you don’t pass a PublisherIdentifier parameter a Global Identifier 00000000-0000-0000-0000-000000000000 will cause throttling frequently as it will calculate resource usage based on all calls that do not pass a PublisherIdentifier. And this includes calls from other tenants as well.

From the error message I see it looks like you are not passing the PublisherIdentifier parameter so the call uses the global PublisherIdentifier 00000000-0000-0000-0000-000000000000.
To fix this you need to pass a Query string PublisherIdentifier=<> to each and every call of the management API.

Re: Microsoft Cloud services Addon stops collecting logs

Azerty728 — Thu, 23 Nov 2017 09:06:36 GMT

It seems a new addon version is on the run on Splunk's side, where it will be possible to change the PublisherID.

Wait & see, stay tuned !

Re: Microsoft Cloud services Addon stops collecting logs

mgrulke — Tue, 28 Nov 2017 21:43:46 GMT

Did they happen to give any possible ETA's on the new add-on update?

Re: Microsoft Cloud services Addon stops collecting logs

pedrolito — Wed, 29 Nov 2017 15:58:50 GMT

Hi splunkers,

Same problem as well. Do you know if this problem was also on oldest addon versions ?

Cheers.

Re: Microsoft Cloud services Addon stops collecting logs

travis_lelle — Wed, 29 Nov 2017 16:23:36 GMT

Splunk support should be able to provide you with the "experimental" version, which is a fixed version that pretty much takes care of the issue.

Re: Microsoft Cloud services Addon stops collecting logs

Azerty728 — Wed, 29 Nov 2017 20:51:54 GMT

They didn't give any info.
I asked them, I filed a case last week about this problem.

I hope we won't be waiting too long.

Cheers

Re: Microsoft Cloud services Addon stops collecting logs

snrnbrem — Fri, 08 Dec 2017 14:56:45 GMT

Hi!
Tagging this as i have the same problem.
Splunk v 6.6.3 and app version 2.0.3.

Azure_AD, Exchange and Sharepoint inputs are getting this error, but the Operational Messages is still functional.

Looking to see if anyone have a good permanent fix to this.

Re: Microsoft Cloud services Addon stops collecting logs

stonecutter_908 — Tue, 19 Dec 2017 12:40:29 GMT

Same issue here. Any update on new rev to the Add On?

Re: Microsoft Cloud services Addon stops collecting logs

pedrolito — Fri, 29 Dec 2017 09:47:19 GMT

Hi Azerty,

Any news about the case you filled ? Or date for beta of new version incoming ?

Thanks a lot

Cheers

Re: Microsoft Cloud services Addon stops collecting logs

Azerty728 — Fri, 29 Dec 2017 10:11:16 GMT

Hi there,

No offical release date. Stay tuned for that. BUT :

I received the addon's patch from Splunk support in order to try (this hotfix is tagged as working by other client, but I didn't get the time to check it on my own).
I encourage you to file a case to Splunk and ask for it if you're impatient enough :D.

I'll update this thread as soon as I've tested it.

Have a nice new year's eve, splunkers 🙂
,Hi there,

No official release date, but you can file a case to Splunk and ask for the hotfix. I received it but didn't have time to test it yet.

I'll update here ASAP if it's working (or not).

Have a nice new year's eve.
Regards

Re: Microsoft Cloud services Addon stops collecting logs

Azerty728 — Fri, 29 Dec 2017 15:05:33 GMT

Good news, it's working !

The patch seems OK : it replaces the PublisherID with our TenantID instead of zeroes, and it works !! I've got plenty of logs now !

I encourage everyone to file a case to Splunk support to ask for the patch, or wait till Splunk releases it officially.

Best regards !

Re: Microsoft Cloud services Addon stops collecting logs

snrnbrem — Thu, 04 Jan 2018 11:45:00 GMT

Confirmed that we also got hands on the intermediate version from Splunk. Installed it and it worked like magic!

Re: Microsoft Cloud services Addon stops collecting logs

stonecutter_908 — Tue, 29 Sep 2020 17:33:16 GMT

I am running the latest beta version of the app given to me by Support and seeing some errors after troubleshooting data that is MIA.

Sample error:
2018-01-11 15:57:16,995 +0000 log_level=ERROR, pid=80730, tid=Thread-6, file=o365_data_collector.py, func_name=_do_safe_index, code_line_no=176 | [input_name="af2a863e-0fb3-462c-80a7-2eddf480771e_Office 365 Management API_Audit.SharePoint" account="TRIM" data="Audit.SharePoint"]Failed to get msg from servers=XXXXXXX, metric=Audit.SharePoint, error=Traceback (most recent call last):
O365GetContentError: [input_name="trim_Office 365 Management API_Audit.SharePoint" account="trim_Splunk_O365_App" data="Audit.SharePoint" proxy_type="http" proxy_rdns="0" proxy_enabled="0" ]Fail to get events of content 20180107230531796018511$20180107230531796018511$audit_sharepoint$Audit_SharePoint, stop this round

Re: Microsoft Cloud services Addon stops collecting logs

pco_support — Tue, 16 Jan 2018 14:56:36 GMT

Hi stonecutter_908,

Could you give the app version you just tested ?

Thx

Cheers

Re: Microsoft Cloud services Addon stops collecting logs

pedrolito — Tue, 16 Jan 2018 15:00:37 GMT

Hi everybody,

I just got the new app from the support team, so ready to test if it's correcting the issue.

Just FYI, the app public release should soon.

Cheers

Re: Microsoft Cloud services Addon stops collecting logs

stonecutter_908 — Tue, 29 Sep 2020 17:40:18 GMT

File itself is Splunk_TA_microsoft-cloudservices-2.0.3.1-3.spl. But it still shows as the GA version of 2.0.3 after install.