Organizations are increasingly choosing to consume software-as-a-service (SaaS) platforms for commodity back-office business applications and end-user collaboration platforms. This is putting increased importance on the Internet egress designs in corporate data networks. Understanding the connectivity flows required to consume these SaaS services is foundational to long term corporate network planning.
One of the most broadly consumed SaaS service is Microsoft’s Office 365 (O365). Microsoft strongly recommends that all O365 customers adopt corporate network designs that have Internet egress close to end-users. For organizations with network topologies using a centralized Internet egress approach, this presents a challenge. Either roll out O365 using a network topology that is not aligned with Microsoft recommendations or delay deployment of O365 until the network topology can be re-designed or until new technologies like SD-WAN can be deployed to allow local Internet breakout for O365 traffic.
Network architects must have a solid understanding of the connectivity patterns of Microsoft’s O365 services to determine if their networks will require topology or technology changes to ensure successful roll outs. In this blog, we explore those connectivity patterns in detail.
Microsoft maintains extensive guidance and documentation to assist organizations with network redesign planning to support Office 365 adoptions.
Microsoft publishes a collection of resources to help network architects and engineers plan Internet connectivity. The following link, O365 Connectivity Principles, provides links to various documentation sources explaining O365 architecture and connectivity guidance.
In addition, Microsoft also publishes lists of locations where they maintain critical components of the O365 architecture. The Azure Edge Node Inventory provides a list of edge nodes that contain front-doors to O365 services such as Exchange Online and SharePoint Online. The O365 Data Storage Locations by Service Page shows the locations of Microsoft Data Centers that house data for the different O365 services.
Microsoft’s guidance stresses the importance of allowing user traffic to route to the Internet as close to the end-user location as possible.
Microsoft has identified end-user experience and performance as critical success factors for the adoption and long-term success of the O365 platform. To support that goal, Microsoft has designed the platform to provide a global network of “service front doors” to help optimize user experience. The objective is to route end-user traffic to the nearest service front door and use optimized high-speed connections across the Microsoft global network backbone to deliver end-user traffic between the service front doors and the appropriate Microsoft O365 data center locations.
The Importance of understanding how Exchange Online connectivity is influenced by DNS configuration
While all O365 connectivity is architected with the goal of getting users as quickly as possible to the Microsoft global network, the Domain Name Service (DNS) resolution configuration for each client will influence where Microsoft will direct Exchange Online connections.
When end-user devices establish connections with the Exchange Online service, the first step in the process is to use DNS to determine an IP address for the hostname outlook.office365.com. The DNS request for outlook.office365.com will be sent to the IP address that the workstation has configured as its DNS resolver. The resolver for most corporate networks is a DNS server inside the corporate network, but smaller organizations may use DNS servers managed by an Internet service provider (ISP). Resolvers perform recursive queries to resolve hostnames for domains controlled by other domain name servers. This involves querying the top-level Internet root domain servers to determine the authoritative name server in the target domain and then making the actual name resolution request to that authoritative name server. Local DNS servers may perform recursive lookups directly or they may use an organizational forwarding server to perform the recursive lookups.
When the server performing the recursive lookup performs the final step, the hostname resolution request, it will send a DNS lookup to the authoritative name server. This is an important element to understand, since the authoritative name server never receives the source IP address of the original endpoint device that is initiating the lookup process. The DNS lookup request that is received at the authoritative name server will contain the source IP address of the DNS resolver or forwarding server.
The name server record for the office365.com domain resolves to an Anycast IP address. This allows Microsoft to maintain DNS servers authoritative for the office365.com domain in each Microsoft network edge location that hosts the Outlook client access front end (CAFE) service. Since the source address of any DNS request uses the source IP address of the resolver, or forwarder, the response will come from the closest Microsoft edge location to the resolver or forwarder. The Microsoft DNS server responding to the outlook.office365.com query uses a load balancing algorithm based on utilization and distance to provide and IP address for a CAFE server located in the same Microsoft Edge node or another close geographic Microsoft Edge node or data center.
If the end-user device and its resolver communicate over the Internet with source IP addresses located in the same geographic region, this results in a connection that minimizes latency from the end-user device to the Exchange Online CAFE server. If however, the resolver or forwarder and the end user device communicate over the Internet with source IP addresses from different geographic regions, then the end-user device could establish sessions with an Exchange Online CAFE server in a different geographic region, possibly leading to undesirable latency characteristics that negatively impact end-user experience. Imagine a scenario where a remote office in Denver has a local Internet egress circuit, but the clients in that office are configured to use a DNS server in the corporate office in Washington DC, which also has an Internet egress circuit. Workstations in the Denver office would connect to Exchange Online using a CAFE server on the east coast, even though there is likely one available in the Denver area.
To demonstrate how this works in practice, Hybrid Pathways configured a test with three ThousandEyes cloud agents located in Oakland, Chicago and Boston. Each cloud agent was instrumented to establish TCP connections with Exchange Online using the outlook.office365.com hostname. Figures 1 through 3 show the network forwarding paths from the ThousandEyes cloud agents to Microsoft Anycast DNS name servers which answered the DNS lookup requests. Figure 4 through 6 show the network forwarding paths from the ThousandEyes cloud agents to the CAFE servers that they were directed to by the Microsoft Anycast DNS servers.
As summarized in Table 1, each testing agent received response from name servers from the nearest Microsoft name server and established TCP connection with geographically close CAFE servers.
Table 1: Exchange Online Connectivity Testing Results
Figure 1: DNS request for outlook.office365.com originating from Oakland
Figure 2: Exchange Online connection originating in Oakland
Figure 3: DNS request for outlook.office365.com originating in Chicago
Figure 4: Exchange Online connection originating in Chicago
Figure 5: DNS request for outlook.office365.com originating in Boston
Figure 6: Exchange Online connection originating in Boston
Be aware that SharePoint Online, OneDrive for Business and Skype for Business connectivity works differently than Exchange Online
Connections to SharePoint Online, OneDrive for Business and Skype for Business work slightly different from Exchange Online. Microsoft maintains service front-doors located at Microsoft edge locations for these services. Anycast IP addresses are used to route users to the closest service front door. All SharePoint Online tenants resolve to the same Anycast IP address, so contoso.sharepoint.com will resolve to the same IP address as acme.sharepoint.com
Using the same testing approach as we did with Exchange Online, three ThousandEyes cloud agents located in Oakland, Chicago and Boston were instrumented to establish TCP connections with the Hybrid Pathways SharePoint Online tenant using the hybridpathways.sharepoint.com hostname. Figures 7 through 9 below show the network path from each of the ThousandEyes cloud agents to the Microsoft edge location that was determined using Anycast routing. Table 2 provides a summary of the edge locations used for each testing source.
Table 2: SharePoint Online Connectivity Testing Results
Figure 7: SharePoint Online connection originating in Oakland
Figure 8: SharePoint Online connection originating in Chicago
Figure 9: SharePoint Online connection originating in Boston
Delivering a high-quality end-user experience for O365 is only partially dependent on minimizing latency between end-user devices and the Internet.
Knowledge of the connectivity patterns for each of the major O365 services is fundamental to any organization that needs to support O365 adoption. Microsoft encourages customers to minimize the latency between end-users and the front doors for the major O365 services.
In addition to developing approaches for allowing O365 traffic to the reach the public Internet as close as possible to the end-user, Microsoft also recommends limiting the amount of latency introduced by security stack elements, specifically Internet proxies that would de-encrypt and re-encrypt traffic to O365 services. Existing proxy environments can be overwhelmed if not scaled out to support the increased load, resulting in increased latency and poor user experience. Media streams for voice and video connections to Skype for Business and Teams must use TCP to establish connections through a proxy instead of connectionless UDP sessions, the normal protocol standard for streaming media communications. This makes it impossible to guarantee SLAs around jitter since TCP must guarantee the delivery of traffic and will buffer traffic during periods of congestion between the end-user and Microsoft media servers.
Microsoft now publishes an API service that allows intelligent network and security devices to query in real time, the IP addresses and URLs used to host critical O365 services such as Exchange Online, SharePoint, OneDrive for Business and Skype for Business/Teams. This allows network infrastructure vendors and security vendors to provide options for routing and inspecting latency sensitive O365 traffic as required. Specifically, instead of configuring network and security devices with static IP address ranges or URLs, these devices can be configured to make queries to the Microsoft REST API service and pull the information directly from Microsoft. This allows network and security devices to avoid stale configurations as Microsoft makes changes to the environment over time.
Following the Microsoft network connectivity guidance will provide for the best-case user experience, but the approach and timing which companies use to redesign their corporate networks in support of O365 adoptions will vary depending on, existing geographic network topology, existing internet egress security stack solutions and utilization profiles for each of the O365 services.
It's also worth noting that not all SaaS providers deliver traffic to their services the same way. For each SaaS service that an organization consumes, network designers need to understand all possible geographic locations that host the services, how often services move around, and what mechanisms are employed (i.e. Anycast, geographic DNS load balancing, etc.) to route users to the front door of those services.
Interested in Learning More?
Hybrid Pathways assists customers with redesigning their corporate networks to facilitate O365 adoption. We leverage our expertise in both network and security architecture to assist the network and security teams of our customers as they evaluate approaches to accommodate the increasing consumption of SaaS applications. Contact us for more information about our services.