All Apps and Add-ons

Ingesting logs from Microsoft Teams

adalbor
Builder

Hey All,

I recently installed/configured the Microsoft Teams Add-on in an attempt to ingest call logs and meeting info from Microsoft Teams.   I have run into an issue I was hoping someone could help with or shed some light on.

Add-On Version: 1.02

Splunk Version: 7.3.4

App is installed on a HF.

I have followed the instructions on the setup and have the Subscription, User Reports, Call Reports and Webhook all setup in the inputs section of the app. It appears though the only thing working is the User Reports. I have granted all of the required permissions in Teams\Azure  per the documentation.

The _internal logs don't give a whole lot of information indicating what the issue might be even with DEBUG logging enabled for the app.

The only thing I am seeing in the logs indicating an issue was this:

127.0.0.1 - splunk-system-user [30/Jun/2020:09:05:36.213 -0500] "GET /servicesNS/nobody/TA_MS_Teams/properties/TA_MS_Teams HTTP/1.1" 404 144 - - - 0ms

And this:

2020-06-30 09:25:43,189 ERROR pid=107176 tid=MainThread file=base_modinput.py:log_error:309 | Could not create subscription: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions

The  documentation also mentions a webook which I am a little confused as to where that webhook resides. Is it in Teams itself or where the app is installed? It seems like the webook is in the app on the HF based on how the documentation reads?

Any help would be greatly appreciated.

Thanks,

Andrew

Labels (3)
0 Karma
1 Solution

jasonabbott
Explorer

Having literally just gone through this, I'll try to help!  What was broken for me (and giving the same headache) sounds like exactly what you're seeing.

If you're getting user reports then your app is correct and the permissions are correct.  What is broken is your either your subscription, webhook, or CDR.  For me, it was the webhook/subscription because they both are interconnected.

First, the Webhook.  The webook has to live on the HF where the Add-on is installed.  The port you give it must be accessible from the public internet (because that's how teams works) and MUST be SSL.  Otherwise nothing will work.  Easiest way to test is to go to the public IP address (from something on the internet) and test https://<webhookName>:<portdefined> and you should get:

{"success": true}

My config looks like this:

 

2020-07-01_06-54-28.png

 

 

 

 

 

 

 

When I got my webhook URL (via https, hostname, port) I get a success.

Once that's done, you configure the Subscription to reference the correct webhook URL.

After that, data should start flowing.

 

Hope this helps, hit me up if you need more help with it!

View solution in original post

Tags (3)

jasonabbott
Explorer

Having literally just gone through this, I'll try to help!  What was broken for me (and giving the same headache) sounds like exactly what you're seeing.

If you're getting user reports then your app is correct and the permissions are correct.  What is broken is your either your subscription, webhook, or CDR.  For me, it was the webhook/subscription because they both are interconnected.

First, the Webhook.  The webook has to live on the HF where the Add-on is installed.  The port you give it must be accessible from the public internet (because that's how teams works) and MUST be SSL.  Otherwise nothing will work.  Easiest way to test is to go to the public IP address (from something on the internet) and test https://<webhookName>:<portdefined> and you should get:

{"success": true}

My config looks like this:

 

2020-07-01_06-54-28.png

 

 

 

 

 

 

 

When I got my webhook URL (via https, hostname, port) I get a success.

Once that's done, you configure the Subscription to reference the correct webhook URL.

After that, data should start flowing.

 

Hope this helps, hit me up if you need more help with it!

Tags (3)

dtrelford
Path Finder

How did you generate your webhook URL? I'm having trouble understanding the format you provided - https://<webhookName>:<portdefined>

My webhook name is "teams_webhook" and the port i chose was 443. Using your provided format, the webhook looks like https://teams_webhook:443 but this isn't a valid URL. 

I also tried https://<publicip>:443/teams_webhook but this also fails.

Later in your post you describe getting a webhook via https, hostname, port; but this also would only work if your hostname is publicly accessible. Any sugggestions?

0 Karma

dtrelford
Path Finder

Ignore my last post! I see now that you already answered this in a previous comment.

0 Karma

atuljha82
Loves-to-Learn Everything

Hello Jason ,

I really appreciate you to give me a response. I have a confusion at here.

1- In Step 1 - Addon mentioned to add webhook name. I give unique name of input in webhook, like Teamswebhook and rest of the field will as is .

2- In Step2 - In teams subscription they ask me to provide the webhook url .Here I am getting confuse -

Either I will provide https://Teamswebhook(That I have add on step1):4444  

OR https://ServerHostname(Where Add-ON is running):4444 .

Could you please help me to know , how I will provide the WebHook url at here.

I will wait for your response.

Thanks

Atul Jha

0 Karma

SinghK
Builder

https://Teamswebhook(That I have add on step1):4444 is what you will add.

Tags (1)
0 Karma

jasonabbott
Explorer

I see the confusion!  The "Name" of the webhook is just for internal use.  The name you use for the "Webhook URL" in the Teams Subscription is https://<serverwhereaddonisrunning>:<port>

Make sure, as I said, that https://<serverwhereaddonisrunning>:<port> is available from the MSFT internet ranges!

nakiamatthews
Explorer

Is your webhook accessible to any public traffic, or were you able to whitelist incoming traffic from Microsoft? I really don't want my Heavy Forwarder exposed to the internet.

0 Karma

adalbor
Builder

I honestly still can't get it to work but can relay our current setup if it helps.

We created an external cert with a specific URL that the webhook would use.

We then ensured the webhook setup in the Splunk app had that URL.

HTTPS inbound to our URL is translated to our specified port at the firewall. If that traffic matches the security policy, it is forwarded on to the F5. The F5 is listening on that port and will pass traffic to the Splunk server on that same port. We do have the Graph API IPs allowed as part of that security policy on the FW.

We can hit the webhook internally via our F5 but still can't get it to work pulling Teams logs.

0 Karma

jasonabbott
Explorer

So, a quick couple of things:

1) Your webhook needs to talk HTTPS, it doesn't need to be on 443.  My test, for instance, is on port 4443.

2) My webhook has allow from all for the moment, but I am working on tightening it down to microsoft's network ranges (see this page: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges). 

If it stops working when I change my ACL, I will post here.

**EDIT**

I updated my ACL to only use the ranges listed in the post above (13.107.6.152/31, 13.107.18.10/31, 13.107.128.0/22, 23.103.160.0/20, 40.96.0.0/13, 40.104.0.0/15, 52.96.0.0/14, 131.253.33.215/32, 132.245.0.0/16, 150.171.32.0/22, 191.234.140.0/22, 204.79.197.215/32) and my CDR's are still flowing correctly.  I will check tomorrow to make sure my user details are still flowing correctly.

 

**EDIT #2**

Verified this morning that all my user data is flowing correctly into the TA.

0 Karma

adalbor
Builder

Yeah my webhook is using 4444.

Glad to hear its still working with good ACL's in place.

Do you have the cert only on your HF and only a FW in between?

0 Karma

jasonabbott
Explorer

That's correct, my cert is on the HF only.  The firewall between the two is a fairly "stupid" setup in that it only allows port/protocol and doesn't do traffic inspection.

adalbor
Builder

Thanks for the super helpful information! Definitely puts me on the right path and kinda confirmed my suspicions.

So you used a cert from an external party and NAT'ed that hostname at your FW?

Tags (1)
0 Karma

jasonabbott
Explorer

Exactly!  Once I did that I have data not only in the Remote Work Insights app, but also in the M365 Teams section. 

Tags (3)
0 Karma

adalbor
Builder

Great! Thanks for the helpful info!

0 Karma

atuljha82
Loves-to-Learn Everything

Hello Jason ,

We are implementing the Microsoft Teams Add-On but here is some confusion if you can help me then it will be great. My heavy forwarder is not running over an internet it is running via http. So I got a ssl certificate from Network Team to install for Microsoft Teams implementation. So below steps I have been taken -

1- I have copied ssl certificate .pem file on to splunk directory-Program Files\Splunk\etc\auth\

2- The above path I have given into Teams Webhook Configuration.

3- In Teams webhook configuration it is also asking the .key file .So here two things-

1- Are you referring Splunk Web key file - to generate new key file

or 

2- I will extract .key file from SSL certificate which I have received from Network Team.

Quick response is highly appreciable.

Thanks

Atul

 

 

 

0 Karma

atuljha82
Loves-to-Learn Everything

Hello All,

Can some-one help me on this for Teams Add-On where Teams plugin have been installed on to Heavy Forwarder and we are getting the user report but we are not getting Call Record data from Teams to Splunk.

While webhook  is live on port 4444 and configure over an Loadbalancer and have allowed the Microsoft Network Ranges in ACL and have verified as well. But still didn't get any data for Call Record.

Quick response is highly appreciable.

 

Thanks

Atul Jha

 

0 Karma

atuljha82
Loves-to-Learn Everything

Hello Jason,

Please do help me at here -

I am facing the issue on subscription log -

2020-12-01 13:17:01,552 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA_MS_Teams/storage/collections/config/?offset=0&search=TA_MS_Teams_checkpointer&count=-1 HTTP/1.1" 200 4535
2020-12-01 13:17:01,553 DEBUG pid=2932 tid=MainThread file=binding.py:new_f:73 | Operation took 0:00:00.003000
2020-12-01 13:17:01,558 DEBUG pid=2932 tid=MainThread file=binding.py:get:677 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA_MS_Teams/storage/collections/data/TA_MS_Teams_checkpoint... (body: {})
2020-12-01 13:17:01,561 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA_MS_Teams/storage/collections/data/TA_MS_Teams_checkpointer/m365_subscription_MS_teams_subscription HTTP/1.1" 404 140
2020-12-01 13:17:01,562 DEBUG pid=2932 tid=MainThread file=base_modinput.py:log_debug:288 | _Splunk_ Getting proxy server.
2020-12-01 13:17:01,562 INFO pid=2932 tid=MainThread file=setup_util.py:log_info:117 | Proxy is not enabled!
2020-12-01 13:17:01,562 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_new_conn:959 | Starting new HTTPS connection (1): graph.microsoft.com:443
2020-12-01 13:17:11,724 DEBUG pid=2932 tid=MainThread file=connectionpool.py:_make_request:437 | https://graph.microsoft.com:443 "POST /beta/subscriptions HTTP/1.1" 400 310
2020-12-01 13:17:11,726 ERROR pid=2932 tid=MainThread file=base_modinput.py:log_error:309 | Could not create subscription: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions
2020-12-01 13:17:11,729 ERROR pid=2932 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\ta_ms_teams\aob_py2\modinput_wrapper\base_modinput.py", line 128, in stream_events
self.collect_events(ew)
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\teams_subscription.py", line 76, in collect_events
input_module.collect_events(self, ew)
File "D:\Program Files\Splunk\etc\apps\TA_MS_Teams\bin\input_module_teams_subscription.py", line 113, in collect_events
raise e
HTTPError: 400 Client Error: Bad Request for url: https://graph.microsoft.com/beta/subscriptions

 

0 Karma

SinghK
Builder

you need to clean the kv store with this command below:

splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer

 

and restart splunk .

 

that should fix the issue. 

0 Karma

adalbor
Builder

@atuljha82 

I had and identical error message in my logs.  This is usually due to Microsoft being unable to communicate with your webhook.  Mine occurred after Microsoft added some new IP's to the graph API and we had to add them to the whitelist in our FW rule.

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...