Cybertelecom
Cybertelecom
Federal Internet Law & Policy
An Educational Project
Internet eXchange Points:

 Dont be a FOOL; The Law is Not DIY

Introduction

"An Internet Exchange Point (IXP) is simply a physical location where different IP networks meet to exchange traffic with each other with copper or fibre cables interconnecting their equipment, usually via one or more Ethernet switches." [The Internet Exchange Point Toolkit & Best Practices Guide, Internet Society, p. 6 (Feb. 25, 2014); The Internet Exchange Point Toolkit & Best Practices Guide, Internet Society, n. 2 (Feb. 25, 2014) ("While there is no formally agreed naming convention, the most commonly used terms are IXP, IX, or exchange point, often shortened to exchange. IXPs are also called INXs, Network Access Points( NAPs), Peering Exchanges, PIC, PIT, and PTT")); Internet Exchange Points, An Internet Society Public Policy Briefing (Oct. 30, 2015) (“Internet exchange points are physical locations where different networks connect to exchange Internet traffic via common switching infrastructures.”)] [Centurylink Peering Policy ("'Interconnection Point' means the place where two IP Networks are connected by dedicated bandwidth for the purpose of exchanging Internet traffic.")] [Suddenlink Peering Policy ("'Interconnection Point' means the place where two IP Networks are connected by dedicated bandwidth for the purpose of exchanging Internet traffic.")]

See also "'Internet Interconnection Points' means the facilities over which traffic is exchanged between the Company’s network that carries Broadband Internet Access Service traffic and (1) peering networks or (2) customers that purchase on-net only services to deliver traffic to and from the Company’s end users over the company’s network." AT&T DirecTV Merger Order App. B.

Typically, the IXP acts as a landlord, renting space to networks and services, providing infrastructure, air conditioning, security, and redundant power.

Advantages: IXPs offer significant advantages: Mook (noting that the motivating factors of increased revenue, cost avoidance, and improved quality of service may be handled by different entities within a network service provider)] [ISOC 2014 ("[T]he primary roles of an IXP are to improve network performance by keeping local internet traffic local and to reduce the costs associated with traffic exchange between networks. This creates a 'virtuous circle.'")]

This reduces costs and increases connection possibilities. BEREC 72, 2011] [BEREC 24-25, 2012] [Nipper]

The success of an IXP is defined by network-effect and scale.

In terms of network-effect, IXPs can become more valuable as more players locate themselves on-site; conversely, IXPs can have difficulty getting off the ground when no or few players have located themselves on-site giving other players no reason to enter the IXP IXPs can have difficulty forming in markets where it is perceived that there are not enough players willing to participate and utilize the IXP. IXPs may also offer special deals to anchor tenants, networks with whom many other networks would want to interconnect, in order to build the value of locating at that IXP. .Canadian Peering Panel (discussing barriers to establishing IXPs)] [Norton, U.S. vs European(noting that IXPs will price discriminate in order to attract anchor tenants who are attractive to other networks in terms of interconnection)] [ISOC, p. 17 2014 (discussing how to get an IXP started)]

Once an IXP gains momentum, scale becomes a strong force. The primary cost for a player is the initial locating at the IXP; once located at an IXP, the cost of establishing interconnection with the N+1 third party using a short, in the building, fiber cross connect, is de minimis. Telegeography Feb. 26, 2015 ("Among the North American markets surveyed by TeleGeography, the median cross-connect price is $300")] [Norton, The U.S. vs. European Internet Exchange Point Models (placing an average recurring cost of a cross connect in a U.S. commercial IXP at $300 / month)]. These interconnections can be quickly negotiated and installed. MLAB p. 3 ("This is usually achieved by linking short wires between routers belonging to one ISP and routers belonging to a second ISP.  Making this physical connection is rarely expensive or tricky - much less so than laying new undersea cable, or digging ditches to install new fiber or other network infrastructure.")] Once located at the IXP, it is easy to grow interconnection.

Estimated IXP Costs (purpose of table is more to show the scale: how much it may cost to get to the IXP and then relatively speaking how little it costs to make the next additional interconnection)

Item
Approx Costs
Fiber to the IXP (major cost)
$1,000,000s / year
Rack Space from IXP (Security, Power, AC, Space)
$1000s / rack / month
Hardware Costs (Routers, Switches, Servers, Chassis, Line Cards)
$1000s / gigabyte
Cross Connects
$300 / Month
Interconnection Fee
  • $0
  • < $1 / mbps
  • ??

    AT&T Mair Declaration, para 42 (listing costs as port costs, cross connects, fiber to the IXP, hardware peering routers and core backbone routers] [Norton Understanding Remote Peering Webinar, Slide 17] [Netflix Florance Declaration at 15 (the cost of a single port as $10,000 which would be amortized over three to five years)]

    Commercial Internet eXchange

    The NSFNET was wildly successful and this created increased demand and opportunity for private commercial packet-switched networks. Many of the companies which successfully won bids to build the ARPANET and the NSFENET leveraged their knowledge to build commercial enterprises. While these networks could interconnect with the NSFNET, the NSFNET restricted use to academic work. Therefore, these new commercial networks, which were not themselves interconnected directly to each other, could not exchange commercial traffic with each other over the NSFNET. The solution was simple: the first commercial exchange point, the Commercial Internet eXchange (CIX). UUNet (acquired by Verizon), PSINET (acquired by Cogent), CERFNET (acquired by AT&T), Sprint, and NSFNET came together in San Jose, California as peers to exchange traffic. Kende p 5]

    Interconnection was on a settlement-free peering basis, and was a reaction against ANS (IBM's NSFNET subcontractor) attempting to establish a monopoly provider for backbone service and charge transit to all other networks.[Greenstein 2015 p. 81] [Srinagesh 143 ("It was this philosophy, together with the inability of the new entrants to obtain interconnection agreements with ANS on terms acceptable to them, that led to the formation of the Commercial Internet eXchange")] [Brock ii 1995 ("Commercial Internet service providers agreed that interchange of traffic among them was of mutual benefit and that each should accept traffic from the other without settlements payments or interconnection charges. The CIX members therefore agreed to exchange traffic on a "sender keep all" basis in which each provider charges it own customers for originating traffic and agrees to terminate traffic for other providers without charge.")] Peering was guided by the following agreement language:

    Member agrees that it shall eXchange network traffic freely with all other CIX members that have access to and use of the CIX NAP ("Participating Members") without payment of settlement fees. Notwithstanding anything to the contrary contained in the preceding sentence, if a connectivity or routing problem caused by a Participating Member is adversely affecting the stability of Member's routing system, then, after delivering advance notice to CIX in a commercially reasonable time period of such problem, Member shall have the right to suspend the exchange of data traffic with such Participating Member until such time as the problem is alleviated. 

    [The Commercial Internet eXchange Association Router Agreement (dated 1997).]

    The role of CIX was important in moving forward commercial interconnection; however, it was almost immediately overshadowed with the establishment of the NAPs. The CIX Router was decommissioned in 2001.

    1990

    "At the same time that privately owned networks started appearing, general commercial activity on the NSFNET was still prohibited by an Acceptable Use Policy. Thus, the expanding number of privately owned networks were effectively precluded from exchanging commercial data traffic with each other using the NSFNET backbone. " [FTC Staff Report 2007 p 18] [Hussain Historic Role CIX 2]

    UUNET, PSInet (William Schrader), and CERFnet (Susan Estrada) meet in order to set up the Commercial Internet eXchange, the first commercial internet peering point, which permitting traffic to be exchange which could not be carried over the NSFNET. [CIX Router TimelinePDF] [CERFnet] The creation of CIX was announced February 14, at the United States Congress Office of Technology Assessment. [Cook pt 3]

    Sprint joined CIX. [Hussain Historic Role CIX 2]

    1991

    Aug 2, the Commercial Internet eXchange was incorporated as a 501(c)(6) trade association. [FTC Staff Report 2007 p 18] [Cook pt 3] [Hussain Historic Role CIX 1] Members of CIX signed onto the CIX Router AgreementPDF which stated in part:

    Member agrees that it shall eXchange network traffic freely with all other CIX members that have access to and use of the CIX NAP ("Participating Members") without payment of settlement fees. Notwithstanding anything to the contrary contained in the preceding sentence, if a connectivity or routing problem caused by a Participating Member is adversely affecting the stability of Member's routing system, then, after delivering advance notice to CIX in a commercially reasonable time period of such problem, Member shall have the right to suspend the exchange of data traffic with such Participating Member until such time as the problem is alleviated.

    [CIX Router AgreementPDF ¶ 2] [Hussain Historic Role CIX 3] Membership in CIX was a flat $10,000. [Cook pt 3]

    The CIX established the business model for the settlement-free exchange of Internet traffic between Network Service Providers. From an engineering perspective that was an important precursor to the Internet interconnection architecture that followed such as the Metropolitan Area Ethernet(MAE) and the NSF sponsored Network Access Points (NAPs) that were established for the transition of the NSFNET traffic to competing service providers that included Sprint, ANS, and MCI.

    CIX router goes online at the Bay Area POP of PSINET. [CIX Router TimelinePDF][Hussain Historic Role CIX 2] While the CIX router would stay online for ten years, it was quickly superceded by the NAPs set up by NSFNET that became the MAEs, and private bilateral interconnection .

    Bilal Chinoy & Timothy J. Salo, Internet Exchanges: Policy-Drive Evolution, in COORDINATING THE INTERNET 325 (Brian Kahin & James H. Keller eds., 1997) 

    1992: ANS (IBM subcontractor serving NSFNET) agrees to interconnect at CIX

    2001: CIX router shutdown. [CIX Router TimelinePDF]

    2005: Smithsonian Museum acquires the CISCO 7500 CIX Router. [Farooq Hussain]

    Federal Internet Exchange

    Milo Medin and Hans Werner Braun propose direct peering for backbone network (not through regional networks) at colo facilities using 10 Mbps ethernet. AMES volunteered as host colo interconnection site. NASA and MILNET were already at AMES. NSF brought a line in for NSFNET. ESNET and ARPANET were also brought to AMES for interconnection. This became known as the Federal Interconnect eXchange (FIX) [Medin Slide 9] (See also CIX) Two FIX exchange points were established, east and west, interconnecting traffic between Dept of Energy, ARPA, NSF, and NASA.

    FIX was managed by the Federal Networking Council and the Federal Engineering Planning Group. [Kesan p 110] [Christopher Jones, Profile: At Home's Milo Medin, WIRED (Jan. 20, 1999)] [Link Letter 1994]

    FIX connected to the NSFNET in June 1989, linking federal internets.

    In 1995, with the retirement of NSFNET and the establishment of the NAPs, the federal agencies recommended that the FIXs be retired and that the federal agencies migrate to, on the East Coast, Sprint's New York NAP, and, on the West Coast, to MAE-West. FEPG Proposal 1995]

    NSFNET & NAPs

    The success of the NSFNET and of new commercial networks that were emerging brought new pressures, both political and market. In 1993, NSF concluded that the NSFNET would be privatized, and that the role of the successful NSFNET would be superseded by the commercial networks. In order to achieve this, the commercial networks would need places to interconnect. [Greenstein 2015 p. 82 (NAPs were modeled on CIX and "helped the commercial Internet operate as a competitive market after the NSFNET shut down.")] Therefore, NSF funded the construction of Network Access Points (NAPs) across the country. In 1994, NSF awards grants for the construction of four Network Access Points:

    Additional Major NAPs include (there are others not listed)

    Aiken p. 14 Discussing mission of NAPs]

    Commercial IXPs

    Initially, major networks collocated at the carrier-operated NAPs and exchanged traffic on a peering basis (consistent with the CIX arrangement). The NAPs served as a catalyst for public Internet growth, resulting in high demand, congestion, and packet loss. Kende 6] [GAO 7-10] [OECD 1998 p. 9 (discussing SAVVIS’ strategy of bypassing NAPs in order to improve network performance)] [Chapin 9 ("Under the terms established by the NFS, a NAP operator was required to provide and operate an interconnection facility on a nondiscriminatory basis, using published pricing and established technical operating specifications.")] [Odlyzko 1998 p. 2 ("traffic through the PacBell NAP... during October 26 and 27, 1997... This NAP was running full blast almost around the clock. As a fraction of the peak rate observed during those two days, the average throughput was 84% on Monday and 80% on Sunday")] SP Gorman, EJ Malecki, The networks of the Internet: an analysis of provider networks in the USA, Telecommunications Policy 24 (2), 113, 117-18 (2000) (“Originally, peering was to be done mainly at network access points (NAPs). But the growth of the Internet caused the NAPs to be overburdened, resulting in a 20-30% packet loss rate… Although the NAPs theoretically provide complete interconnectivity of the Internet, they are both public and congested.”)] [Noam 65 ("NAPS began to cause bottlenecks")]

    As backbones sought to avoid the congested public NAPs, new IXPs and business models emerged: carrier-neutral IXPs. This could be either a commercial or non-profit collocation business, providing an interconnection space that was not tied to any specific carrier. BEREC p. 24 2012] [Chapin ("These exchanges provided a framework that allowed multiple providers of different sizes, scopes, and operating philosophies, serving the same or different markets, to interconnect in ways appropriate to each.")] [Norton]

    Institutional models for IXPs: Mwangi] [BEREC p. 14 2014] ISOC 2014 p. 14 (IXPs "fall roughly into five categories: (1) Non profits (2) Industry associations (3) Carrier-neutral commercial (4) university or government, or (5) informal)]

    Except of Carrier Hosted IXPs, most IXPs as a part of their business plan do not compete with their members or customers. For example, although IXPs with multiple POPs may have fiber connecting their own POPs, they may be disinclined to sell commercial access to that fiber and thereby compete with their transit members or customers. ISOC 2014 p. 16]

    Locations

    Interconnection generally takes place at Internet eXchange Points (IXPs). The primary regional interconnection locations where the Tier 1 backbones interconnect include:

    • Ashburn, VA
    • New York
    • Chicago
    • Atlanta
    • Dallas
    • San Jose
    • Los Angeles, and
    • Seattle.

    Norton, Evolution of US Ecosystem] [TWT ("In the United States, at a minimum, the Requester must have a backbone node in each of the following eight geographic regions: Northeast; Mid-Atlantic; Southeast; North Central; South Central; Northwest; Mid-Pacific; and Southwest.")] [Centurylink(Centurylink requires the exchange of traffic in at least 10 major metropolitan statistical areas)] [XO (requiring interconnection at 9 locations including as possibilities the 8 locations listed above, Denver, Miami and international locations)] [AT&T (listing the above 8 peering locations as well as Miami)] [Comcast (listing 8 peering locations as well as Denver and Miami)] [Level 3] [Compare Hurricane Electric Peering Requirements (listing numerous peering locations)]

    These major peering cities track the original locations of interconnection established by NSF with the NAPs, FIXes, and CIX.

    Both Denver and Miami are also significant interconnection locations. The major interconnection cities are largely a product of the originally interconnection cities designated by NSFNET. Backbones which peer generally will interconnect at multiple major interconnection cities.

    According to Packet Clearing House, there are 80 IXPs in the United States. [PCH] [Telegeography Internet Exchange Map] [Public Exchange Point Search, PeeringDB]IXPs can be located in regional and local locations. Traffic exchanged at IXPs may be more localized, implementing MEDs / Cold Potato routing, with metro destined traffic being delivered only to a metro network.

    The number of IXPs worldwide has grown steadily.

    © Cybertelecom ::