topic Ingesting logs from Microsoft Teams in All Apps and Add-ons

Ingesting logs from Microsoft Teams

adalbor — Tue, 30 Jun 2020 15:07:52 GMT

Hey All,

I recently installed/configured the Microsoft Teams Add-on in an attempt to ingest call logs and meeting info from Microsoft Teams. I have run into an issue I was hoping someone could help with or shed some light on.

Add-On Version: 1.02

Splunk Version: 7.3.4

App is installed on a HF.

I have followed the instructions on the setup and have the Subscription, User Reports, Call Reports and Webhook all setup in the inputs section of the app. It appears though the only thing working is the User Reports. I have granted all of the required permissions in Teams\Azure per the documentation.

The _internal logs don't give a whole lot of information indicating what the issue might be even with DEBUG logging enabled for the app.

The only thing I am seeing in the logs indicating an issue was this:

127.0.0.1 - splunk-system-user [30/Jun/2020:09:05:36.213 -0500] "GET /servicesNS/nobody/TA_MS_Teams/properties/TA_MS_Teams HTTP/1.1" 404 144 - - - 0ms

And this:

2020-06-30 09:25:43,189 ERROR pid=107176 tid=MainThread file=base_modinput.py:log_error:309 | Could not create subscription: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions

The documentation also mentions a webook which I am a little confused as to where that webhook resides. Is it in Teams itself or where the app is installed? It seems like the webook is in the app on the HF based on how the documentation reads?

Any help would be greatly appreciated.

Thanks,

Andrew

Re: Ingesting logs from Microsoft Teams

jasonabbott — Wed, 01 Jul 2020 11:58:41 GMT

Having literally just gone through this, I'll try to help! What was broken for me (and giving the same headache) sounds like exactly what you're seeing.

If you're getting user reports then your app is correct and the permissions are correct. What is broken is your either your subscription, webhook, or CDR. For me, it was the webhook/subscription because they both are interconnected.

First, the Webhook. The webook has to live on the HF where the Add-on is installed. The port you give it must be accessible from the public internet (because that's how teams works) and MUST be SSL. Otherwise nothing will work. Easiest way to test is to go to the public IP address (from something on the internet) and test https://<webhookName>:<portdefined> and you should get:

{"success": true}

My config looks like this:

When I got my webhook URL (via https, hostname, port) I get a success.

Once that's done, you configure the Subscription to reference the correct webhook URL.

After that, data should start flowing.

Hope this helps, hit me up if you need more help with it!

Re: Ingesting logs from Microsoft Teams

adalbor — Wed, 01 Jul 2020 13:09:57 GMT

Thanks for the super helpful information! Definitely puts me on the right path and kinda confirmed my suspicions.

So you used a cert from an external party and NAT'ed that hostname at your FW?

Re: Ingesting logs from Microsoft Teams

jasonabbott — Wed, 01 Jul 2020 14:25:06 GMT

Exactly! Once I did that I have data not only in the Remote Work Insights app, but also in the M365 Teams section.

Re: Ingesting logs from Microsoft Teams

adalbor — Wed, 01 Jul 2020 14:34:19 GMT

Great! Thanks for the helpful info!

Re: Ingesting logs from Microsoft Teams

nakiamatthews — Thu, 30 Jul 2020 14:38:43 GMT

Is your webhook accessible to any public traffic, or were you able to whitelist incoming traffic from Microsoft? I really don't want my Heavy Forwarder exposed to the internet.

Re: Ingesting logs from Microsoft Teams

adalbor — Thu, 30 Jul 2020 15:36:37 GMT

I honestly still can't get it to work but can relay our current setup if it helps.

We created an external cert with a specific URL that the webhook would use.

We then ensured the webhook setup in the Splunk app had that URL.

HTTPS inbound to our URL is translated to our specified port at the firewall. If that traffic matches the security policy, it is forwarded on to the F5. The F5 is listening on that port and will pass traffic to the Splunk server on that same port. We do have the Graph API IPs allowed as part of that security policy on the FW.

We can hit the webhook internally via our F5 but still can't get it to work pulling Teams logs.

Re: Ingesting logs from Microsoft Teams

jasonabbott — Fri, 31 Jul 2020 13:21:26 GMT

So, a quick couple of things:

1) Your webhook needs to talk HTTPS, it doesn't need to be on 443. My test, for instance, is on port 4443.

2) My webhook has allow from all for the moment, but I am working on tightening it down to microsoft's network ranges (see this page: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges).

If it stops working when I change my ACL, I will post here.

**EDIT**

I updated my ACL to only use the ranges listed in the post above (13.107.6.152/31, 13.107.18.10/31, 13.107.128.0/22, 23.103.160.0/20, 40.96.0.0/13, 40.104.0.0/15, 52.96.0.0/14, 131.253.33.215/32, 132.245.0.0/16, 150.171.32.0/22, 191.234.140.0/22, 204.79.197.215/32) and my CDR's are still flowing correctly. I will check tomorrow to make sure my user details are still flowing correctly.

**EDIT #2**

Verified this morning that all my user data is flowing correctly into the TA.

Re: Ingesting logs from Microsoft Teams

adalbor — Thu, 30 Jul 2020 16:11:44 GMT

Yeah my webhook is using 4444.

Glad to hear its still working with good ACL's in place.

Do you have the cert only on your HF and only a FW in between?

Re: Ingesting logs from Microsoft Teams

jasonabbott — Thu, 30 Jul 2020 16:20:25 GMT

That's correct, my cert is on the HF only. The firewall between the two is a fairly "stupid" setup in that it only allows port/protocol and doesn't do traffic inspection.

Re: Ingesting logs from Microsoft Teams

atuljha82 — Mon, 03 Aug 2020 13:27:23 GMT

Hello Jason ,

I really appreciate you to give me a response. I have a confusion at here.

1- In Step 1 - Addon mentioned to add webhook name. I give unique name of input in webhook, like Teamswebhook and rest of the field will as is .

2- In Step2 - In teams subscription they ask me to provide the webhook url .Here I am getting confuse -

Either I will provide https://Teamswebhook(That I have add on step1):4444

OR https://ServerHostname(Where Add-ON is running):4444 .

Could you please help me to know , how I will provide the WebHook url at here.

I will wait for your response.

Thanks

Atul Jha

Re: Ingesting logs from Microsoft Teams

jasonabbott — Mon, 03 Aug 2020 13:34:05 GMT

I see the confusion! The "Name" of the webhook is just for internal use. The name you use for the "Webhook URL" in the Teams Subscription is https://<serverwhereaddonisrunning>:<port>

Make sure, as I said, that https://<serverwhereaddonisrunning>:<port> is available from the MSFT internet ranges!

Re: Ingesting logs from Microsoft Teams

dtrelford — Fri, 07 Aug 2020 15:00:34 GMT

How did you generate your webhook URL? I'm having trouble understanding the format you provided - https://<webhookName>:<portdefined>.

My webhook name is "teams_webhook" and the port i chose was 443. Using your provided format, the webhook looks like https://teams_webhook:443 but this isn't a valid URL.

I also tried https://<publicip>:443/teams_webhook but this also fails.

Later in your post you describe getting a webhook via https, hostname, port; but this also would only work if your hostname is publicly accessible. Any sugggestions?

Re: Ingesting logs from Microsoft Teams

dtrelford — Fri, 07 Aug 2020 15:04:08 GMT

Ignore my last post! I see now that you already answered this in a previous comment.

Re: Ingesting logs from Microsoft Teams

atuljha82 — Mon, 12 Oct 2020 17:30:32 GMT

Hello Jason ,

We are implementing the Microsoft Teams Add-On but here is some confusion if you can help me then it will be great. My heavy forwarder is not running over an internet it is running via http. So I got a ssl certificate from Network Team to install for Microsoft Teams implementation. So below steps I have been taken -

1- I have copied ssl certificate .pem file on to splunk directory-Program Files\Splunk\etc\auth\

2- The above path I have given into Teams Webhook Configuration.

3- In Teams webhook configuration it is also asking the .key file .So here two things-

1- Are you referring Splunk Web key file - to generate new key file

2- I will extract .key file from SSL certificate which I have received from Network Team.

Quick response is highly appreciable.

Thanks

Atul

Re: Ingesting logs from Microsoft Teams

atuljha82 — Mon, 30 Nov 2020 19:58:04 GMT

Hello All,

Can some-one help me on this for Teams Add-On where Teams plugin have been installed on to Heavy Forwarder and we are getting the user report but we are not getting Call Record data from Teams to Splunk.

While webhook is live on port 4444 and configure over an Loadbalancer and have allowed the Microsoft Network Ranges in ACL and have verified as well. But still didn't get any data for Call Record.

Quick response is highly appreciable.

Thanks

Atul Jha

Re: Ingesting logs from Microsoft Teams

atuljha82 — Tue, 01 Dec 2020 20:58:47 GMT

Hello Jason,

Please do help me at here -

I am facing the issue on subscription log -

2020-12-01 13:17:01,552 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA_MS_Teams/storage/collections/config/?offset=0&search=TA_MS_Teams_checkpointer&count=-1 HTTP/1.1" 200 4535
2020-12-01 13:17:01,553 DEBUG pid=2932 tid=MainThread file=binding.py:new_f:73 | Operation took 0:00:00.003000
2020-12-01 13:17:01,558 DEBUG pid=2932 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA_MS_Teams/storage/collections/data/TA_MS_Teams_checkpointer/m365_subscription_MS_teams_subscription (body: {})
2020-12-01 13:17:01,561 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA_MS_Teams/storage/collections/data/TA_MS_Teams_checkpointer/m365_subscription_MS_teams_subscription HTTP/1.1" 404 140
2020-12-01 13:17:01,562 DEBUG pid=2932 tid=MainThread file=base_modinput.py:log_debug:288 | _Splunk_ Getting proxy server.
2020-12-01 13:17:01,562 INFO pid=2932 tid=MainThread file=setup_util.py:log_info:117 | Proxy is not enabled!
2020-12-01 13:17:01,562 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_new_conn:959 | Starting new HTTPS connection (1): graph.microsoft.com:443
2020-12-01 13:17:11,724 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://graph.microsoft.com:443 "POST /beta/subscriptions HTTP/1.1" 400 310
2020-12-01 13:17:11,726 ERROR pid=2932 tid=MainThread file=base_modinput.py:log_error:309 | Could not create subscription: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions
2020-12-01 13:17:11,729 ERROR pid=2932 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\ta_ms_teams\aob_py2\modinput_wrapper\base_modinput.py", line 128, in stream_events
self.collect_events(ew)
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\teams_subscription.py", line 76, in collect_events
input_module.collect_events(self, ew)
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\input_module_teams_subscription.py", line 113, in collect_events
raise e
HTTPError: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions

Re: Ingesting logs from Microsoft Teams

adalbor — Tue, 15 Dec 2020 18:12:45 GMT

@atuljha82

I had and identical error message in my logs. This is usually due to Microsoft being unable to communicate with your webhook. Mine occurred after Microsoft added some new IP's to the graph API and we had to add them to the whitelist in our FW rule.

Re: Ingesting logs from Microsoft Teams

SinghK — Mon, 07 Jun 2021 11:21:41 GMT

https://Teamswebhook(That I have add on step1):4444 is what you will add.

Re: Ingesting logs from Microsoft Teams

SinghK — Mon, 07 Jun 2021 11:29:30 GMT

you need to clean the kv store with this command below:

splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer

and restart splunk .

that should fix the issue.