Level 3 Blog

Analysis of the Mirai botnet: [1] This is worth reading. It has links to copies of the code and names the known control servers. Quite a bit is known now about how this thing works. The bots talk to control servers and report servers. The attacker appears to communicate with the report servers over Tor. [1] http://blog.level3.com/security/grinch-stole-iot/

There is nothing reasonable about this. You pay Comcast for bandwidth. It shouldn't matter how much data you're using as long as you're not exceeding that bandwidth. If I pay for 250mbps, I should be able to download 24/7 (assuming I did the math right, that's 83.7 TB). Comcast isn't storing the bits, they're just moving them. This is simply their way of avoiding having to upgrade their peering arrangements (which are already abysmal). Level3 has a great blog post explaining a similar issue (http://blog.level3.com/open-internet/observations-internet-middleman/)…

The transit providers have pretty much all agreed with Netflix's side of the story to the best of my knowledge. For example: http://blog.level3.com/open-internet/observations-internet-middleman

1) Because they aren't a last-mile carrier, but instead a backbone provider and thus don't have chips on the table. 2) As trevelyan said, they operate around the world, and state that this situation, and mindset of last-mile carriers, is unique to the US. 3) One would think a backbone provider would share the same views as the last-mile carriers, since it would mean more money for them too. However, they state that big content providers such as Netflix are very much so not a problem, but instead it's the last-mile carrier's refusal to make necessary network infrastructure upgrades to augment capacity. This information is available on their blog. This post in particular worth a read: http://blog.level3.com/global-connectivity/chicken-game-played-child-i…

If you'd like a different perspective regarding the quality of your broadband connection and the reasons for congestion this is a good read that basically says ATT at least does not provide enough connectivity from the Level 3's optical backbone to their regional networks. I suspect Verizon is probably the same. http://blog.level3.com/global-connectivity/chicken-game-played-child-i… Here's a letter Level3 wrote to ATT on the subject [pdf]. http://www.level3.com/~/media/Assets/legal/cicconi.pdf

"From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement." I would argue that point. I pay for Internet access, not "All of the Internet MINUS Netflix" access. As a result my ISP SHOULD (doesn't mean they will) ensure that all it's peering connections are of sufficient capacity to provide me with whatever bandwidth I choose to use up until I run into the rate limits for the plan I'm on. My ISP sells me 25/5 I think so the idea that I can't get 3Mbps worth of Netflix is suspect. Of course you have to consider that when you're not saturating your last-mile connection it could be a bottleneck anywhere. If it's 15 hops from me to a server if any of those 15 hops is saturated, that's the limiting factor no matter how much capacity there is before or after. That's why closing a single lane on a freeway for as little as 10 feet can really screw up traffic. You have to merge down and deal with reduced capacity even though it's not miles and miles of lane closure. What we've been told via one of Netflix's backbone providers (Level 3) is that Netflix isn't the slowdown, the backbone provider isn't the slowdown, but the ISPs are the slowdown. How can they say this with certainty? Because they're peered with various ISPs, the connections are saturated, and the ISPs refuse to upgrade their side of the peering connection unless they're paid. http://blog.level3.com/global-connectivity/observations-internet-middl… That might sound…

> it's a 20:1 disparity in traffic in vs traffic out. Are you talking about on this graph? http://blog.level3.com/wp-content/uploads/2014/05/route_info_1.jpg Because all the numbers I see are showing about a 5:1 or 6:1 imbalance. > ...transit providers are starting to see their industry be squeezed by the big ISPs. No. > ...why do transit providers even exist? Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet). Take another look at the L3 network map from the blog post: http://blog.level3.com/wp-content/uploads/2014/05/network_map.jpg If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP. > Settlement-free peering assumes that bandwidth usage is roughly symmetric No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile. Comcast empirically does not care…

http://blog.level3.com/security/attack-of-things/ Getting manufacturers to patch, and users to update these embedded linux devices is going to be pretty hard

Level3 had an interesting blog post on this a while back: http://blog.level3.com/open-internet/observations-internet-middleman/ Basically certain ISPs refuse to increase connection capacity with backbone providers like level3. Instead they (Comcast) is trying to get Netflix to connect directly with them and charge them a premium in doing so.

Squirrels accounted for 28% of fiber cuts in 2010 for Level 3 Communications.[1] [1] http://blog.level3.com/level-3-network/the-10-most-bizarre-and-annoyin…

US broadband interconnects with the rest of the network are pretty terrible. Reference: those name-and-shame articles a while back from a network company (Level 3, from memory?) that produced hard numbers on how little $ US cable/DSL internet companies were willing to put into improving their capacity at the peer link. [Edit] Yup, Level 3. I thought they were interesting because hard numbers at this level seem difficult to come by. Here are the articles from 2014: http://blog.level3.com/open-internet/chicken-game-played-child-isps-in… http://blog.level3.com/open-internet/observations-internet-middleman/

How can this go on so long and not get solved? Because Comcast is probably doing it on purpose. http://blog.level3.com/open-internet/observations-internet-middleman/ http://arstechnica.com/business/2015/05/comcast-ends-an-interconnectio…

Level3's VP disagrees: http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-dns/ "Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

Last I checked it was the consumer ISPs that were stalling with peering upgrades according to multiple transit providers http://blog.level3.com/open-internet/verizons-accidental-mea-culpa/ https://news.ycombinator.com/item?id=9264381

It certainly could be either side of the interconnect that's causing the problem. The important insight is that no ISP is uniformly bad at either end, which means that the problem is not at either end, but in the middle. So this study has found congestion in the core (at the interconnects), precisely where it is not supposed to be, and has found that congestion to be so bad that it drives performance to sub-broadband levels. Cogent and L3 are ISPs as well, so I think saying "ISPs are failing to provide broadband speeds" is fair. However, to add context, L3 had a blog post a while back explaining that they are standing ready to add more capacity for cheap and that the congestion is, in at least one instance (in LA), Verizon's fault. http://blog.level3.com/open-internet/verizons-accidental-mea-culpa/

He's referring to http://blog.level3.com/open-internet/verizons-accidental-mea-culpa/. Verizon will happily take your money to give you 500 Mbps to anything on their own network, but for anything on the internet at large, be prepared to wait during peak traffic hours.

Here's a link to one of the blog posts: http://blog.level3.com/global-connectivity/verizons-accidental-mea-cul…

This article is readable and explains the basics of 'transit' (I think peering is the correct term in this case). http://blog.level3.com/global-connectivity/observations-internet-middl…

I had this same experience with Comcast. See also Level 3's blog post calling Verizon and other ISPs out on this BS: http://blog.level3.com/global-connectivity/observations-internet-middl…

Heck, even I write mails to my ISP and tell them which interconnect points they're having issues at which a simple traceroute/mtr show me. And I mostly get positive feedback. With a bit more sophisticated tools, you'll get more confidence. Or if you're Netflix, I'd assume you can just call up level3, ask them to resolve the transit issues you're having with Verizon, and get feedback and reports about it being verizon not upgrading their interconnect to level3. I'd not be terribly surprised if verizon is one of the 6 peers mentioned at http://blog.level3.com/global-connectivity/observations-internet-middl…

The issue is that (as shown by Level3 [1]) the congestion isn't really at the intranet but within regional ISP networks. The regional ISPs (Comcast/TWC etc) realized that actually not spending money to augment their networks to accommodate for the traffic can be more profitable than actually providing a better service to their customers. When the traffic on their network is congested, they can force companies like Netflix to actually run a fiber directly to their data center not only they can have Netflix pay for it (and therefore they don't have to pay anything to their uplink (such as Level3 and others), they also will receive money from these companies in order to not throttle their traffic. And they can do all that only because they have regional monopoly, and users don't have anywhere else to go. [1] http://blog.level3.com/global-connectivity/observations-internet-middl…

> Just make it illegal to sell more capacity than you have and the problem is solved. No. The result would be that you would pay 100 times more for your Internet connection. Or, alternatively, that you cannot get a guaranteed bandwidth, but you pay per GB. This article: http://blog.level3.com/global-connectivity/observations-internet-middl… states that Level3 has 13,600 Gbps of capacity with its peers. Your solution would mean that Level3's peers can only sell 13.6 million 1 Mbps connections without overselling their bandwidth. Even if Level3's peers have several other Internet wholesale agreements -- let's say they actually have 5 times that capacity (most likely an overstatement) -- all Level3's peers would only be able to sell 68 million 1 Mbps connections. That would be some expensive 1 Mbps connections!

It maybe not be linear, but if Netflix starts sending 2x traffic, they will pay something like 1.8x-2x. That's not changing, and nobody is disputing this. Are you arguing that Netflix's costs to scale traffic into Comcast are commensurate with Comcast's costs to scale the delivery of that traffic to consumers? I don't think that's true. Didn't Level 3 just blog[1] that they largely operate on a settlement-free basis? Prior to their deal with Comcast, Netflix may have been paying Level 3, and Level 3 may have been paying Comcast, but neither could have been paying the order-of-billions Comcast would need to spend to support, say, doubled Netflix primetime traffic. [1] http://blog.level3.com/global-connectivity/observations-internet-middl…

The problem is that america is almost a monopoly, no other country really has these problems[0] [0] http://blog.level3.com/global-connectivity/observations-internet-middl…

The bilateral arrangement here is between the ISP and an internet backbone - in this case, likely Level3. Level3 has stated that they are willing to split the cost of upgrading the link. Verizon is unwilling to do so. It's also useful to note that the cost of upgrading a link is tiny in the terms of these companies - on the order of $10-20k[1]. Netflix already pays to have sufficient capacity at their end of the connection; it is the ISP's responsibility to do the same, since their customers are paying to access whatever internet content they want. It is unreasonable to expect Netflix to pay for a separate, unnecessary connection. That's more like if the Buffalo airport decided they were getting too many flights from LA, so if anyone else from LA wanted to fly to Buffalo, they would have to pay for the addition of their own private terminal. It's not the way things are normally done, and it doesn't make sense. [1] http://blog.level3.com/global-connectivity/observations-internet-middl…

You might like to read Level3's take on it. A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers. That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers. http://blog.level3.com/global-connectivity/observations-internet-middl…

It's not just Netflix, it's also YouTube (http://www.google.com/get/videoqualityreport/), and Level3 (http://blog.level3.com/global-connectivity/observations-internet-middl… and you can be sure those aren't the exception. Instead of fulfilling their obligations to provide advertised bandwidth to their paying customers (who only pay them in order to access the content provided by someone else), the last-mile providers are using their customers as hostages while they negotiate ransom from the content providers. While they also report record profits (sending their customer's payments back to stockholders instead of into the network), because the economics of their business are compelling on both the consumer side (low expectations, minimal support costs, essentially no competition) and on the technology side (network technologies increase capacity-per-$ as fast or faster than traffic grows). Oh, also, 99% of the last-mile providers operate a sanctioned monopoly that has let them build and grow their networks with essentially guaranteed income from rate-payers and essentially no competition. Hence adding Internet is relatively low-cost and highly-profitable, but apparently not profitable enough.

Are you talking about this article? http://blog.level3.com/global-connectivity/observations-internet-middl… > A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers. > That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. So I'm reading it with my republican hat on, and I'm thinking: why is this any business of the government's? This sounds like a contractual dispute between two network companies over upgrades. Why should the government get involved in this? The "consumer satisfaction" angle at the end might be more compelling, but more to a Ralph Nader type than even a Bill Clinton type much less a George Bush type. And the Ralph Nader types raised these objections in 1996 in opposition to telecoms deregulation, and they lost over a bi-partisan effort. You can raise the specter…

I'm fairly sure that the "4.2.2.x was never meant to be public" line is a myth. Though the NANOG thread cited in that post is good historical background, it is contradicted by more modern sources: "[...] DNS infrastructure is largely split into two types; open (public) and closed (private). Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required." - Mark Taylor, VP at Level3 http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-dns/ I can't find any cite where anyone else who I would consider a reliable source in the DNS world (Vixie, &c) repeat this claim. To the contrary, Level 3 is often grouped with Google, OpenDNS and others in discussions of open public resolvers [1][2][3], and those in the know never seem to speak up and say otherwise in these discussions. That being said, I have absolutely no personal knowledge on any of this. [1] http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_MAAWG_India… [2] http://markmail.org/message/gh7f2wvfbn5mpvuq [3] http://www.circleid.com/posts/87143_dns_not_a_guessing_game/#4234