ip location and microsoft/google

[sweiland]

Hello Everyone,

We are currently working on exchange logs (IIS), and trying to detect abnormal traffic from different countries for a unique user, which seemed fairly simple.

The main problem is that, as we found out, most of Outlook for mobile data is proxied through Microsoft network (no comment on data security...), so it could come from several location for a single user (ireland, usa, etc).

To set up the dashboard, we want to exclude (for now), every IP that cidr match the microsoft network (will do the same for other cloud providers). For this, we wanted a way to insert the networks in a lookup table and add a field to the search request if the IP is owned by a cloud provider.

Here is an example of the MS network (might not be exhaustive though..):

https://docs.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-world...

Any idea  on how to:

- Add simply all this garbage to a lookup file containing the networks + provider label (here Microsoft)

- Make the lookup and then add a field if the ip is in the lookup table (like a field "Cloud based IP" which contains the Provider)

As a result, we will be able to filter out Microsoft/Google/Amazon from the anomalies...

Would help a lot, and hope it will help other that are trying to get a better understanding of external outlook connections..

[sweiland]

Very interesting, will be interesting to inject all these in a lookup table to start with

Thing is it will be also difficult to see regular traffic from AWS/Google/MS (the famous proxy from the email app) from an attacker using Amazon cloud services trying to connect, what a pain to achieve something that seemed to simple at first 🙂

[nickhills]

Its not too big a job, let me know if you want to try and i'll see if i can help

[sweiland]

Thanks a lot for pointers

We did manage to extract the json and parse/convert it to produce a simple CSV lookup in Splunk.

We have 2 fields and it is quite easy (Ip_Range/Network_Name).

Only problem we have so far is that despite using the "CIDR(Ip_Range)" match type in the lookup definition, it doesnt match any IP which has the subnet behind.

For example:

158.64.79.14 would match 158.64.79.14 in the csv, but not 158.64.79.14/32 (do not even ask for network ranges)

Is there anything to do about this ?

Query is quite simple:

index="msexchange" | stats count by xff | lookup cloudproviders_lookup Ip_Range as xff OUTPUTNEW Network_Name

[skyelowryvancit]

Just what I am looking for!

Can you share any of your code for the creating the lookup table from the JSON?

[sweiland]

Sure, I will dig on the scripts tomorrow.

Short story: from the JSON I retrieve (wget/curl), I create an ordered CSV file which is dynamic and imported into splunk. I can give you some lines to render the JSON into CSV if that may be useful?

I might even make a short LinkedIn article about that because the more I talk about this, the more people are surprised to learn that outlook mobile traffic is proxified by MS (so it is fairly difficult to say if an ip source is suspicious or not).

What bothers me is that if someone uses outlook mobile, you are not even able to know the actual location of the fellow (he might use a VPN also yeah, but at least not merged in the MS ip...)

[skyelowryvancit]

Even the code you have for that move from the JSON to CSV would be great. I think I might try to make an app that does this activity, and updates a lookup for all the cloud providers.

Yes, MS does make it confusing often, there might be fewer use cases than I imagine with the data set. Good to start digging though.

If you do a linkedin article post the link 🙂 

[sweiland]

Nothing too fancy... (and not optimized for bash gurus :p)

#!/bin/bash
TIMESTAMP=$(date '+%Y%m%d%H%M%S')
URL="https://endpoints.office.com/endpoints/worldwide?noipv6&ClientRequestId=b10c5ed1-bad1-445f-b386-b919946339a7"
OUTPUTFILE="${TIMESTAMP}_MappingAllNetwork"

echo "Retrieving Microsoft IPs:"
curl "${URL}" > "${OUTPUTFILE}_raw"
jq . "${OUTPUTFILE}_raw" > "${OUTPUTFILE}_json" && rm "${OUTPUTFILE}_raw"
egrep -o '[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+/[0-9]+' "${OUTPUTFILE}_json" | sort -un -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | grep -o '^[^#]*' | sed 's/$/,EXTERNAL,WIRED,CLOUD PROVIDERS/' > "${OUTPUTFILE}_sorted" && rm "${OUTPUTFILE}_json"

We produce a csv as follows: IpRange,Zone,Type,Site

So this is why you see "EXTERNAL, WIRED, CLOUD PROVIDERS" 🙂

[sweiland]

We found out, it was because the directive in the lookup was CIDR([Ip_Range]) and not CIDR(Ip_Range)

[nickhills]

I found this list too - IPs of all 365 services:

https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a...

[nickhills · Accepted solution]

I know AWS makes a json file of all current IP addresses available for download: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

A quick search has not turned up the same for Microsoft/Google.

In the past I have had a job to pull and ingest the AWS file and then transform it into a table (and thus a lookup) for this very purpose.

Update:

Google provided list for "Google Services": https://www.gstatic.com/ipranges/goog.json

GitHub project for obtaining GCP addresses https://gist.github.com/n0531m/f3714f6ad6ef738a3b0a

Manual downloads for MS Azure addresses: https://azure.microsoft.com/en-us/updates/service-tag-discovery-api-in-preview/