What looks like a major outage has hit T-Mobile while Verizon, AT&T, and Sprint this afternoon are also seeing issues. Hundreds of thousands of T-Mobile customers are saying they can’t make calls and texts with some having no Internet connection too. The problems for all four carriers appear to be nationwide.
Update: Response from T-Mobile, AT&T, and Verizon below. As it turns out, it’s just a T-Mobile issue.
Update 6:55 p.m. PT: T-Mobile is still working on a fix. Meanwhile, Cloudflare founder Matthew Prince has a helpful thread explaining that there has not been a big DDoS attack on the US. Instead, it was a mishap as T-Mobile was “making some changes to their network configurations today. Unfortunately, it went badly.” Read the full thread from Prince below for a behind the scenes look at what happened today and how it looked like a much broader problem than it actually is.
Update 7:52 p.m. PT: The most recent word from T-Mobile’s President of Technology, Neville Ray on Twitter: “My team is working hard to fix it and I will let you know as soon as the issue is resolved. Thank you for your patience.”
Update 11:53 p.m. PT: Ray shared that T-Mobile service has been restored. Just before fixing the problem, T-Mobile CEO Mike Sievert shared in a blog post some vague details about the outage: “This is an IP traffic related issue that has created significant capacity issues in the network core throughout the day.” As mentioned above, insight from Matthew Prince sheds more light on the situation and how it was the result of an accident at T-Mobile, not a DDoS attack as well as how it looked like many carriers/services were impacted even though it was just a T-Mobile issue.
This afternoon T-Mobile users began seeing the service down for making calls and texts. At the time of writing, over 100,000 people have logged issues on Downdetector.com and also shared the problems they’re having on Twitter. Some are seeing their Internet connection still working with calls and texts being the issue while some are seeing all service from T-Mobile down.
Reports are coming from all around the US for the T-Mobile outage. Verizon, AT&T, and Sprint users are also reporting issues today but the biggest impact to service seems to be with T-Mobile so far.
T-Mobile’s Metro service is also seeing downtime with one customer being told by the company that the carrier is doing an “upgrade” of some sort. But that’s just one story from one customer service rep. There appears to be a much bigger problem happening.
The curious thing is that the map of reported issues across all four major US carriers appears similar, with affected areas spanning most of the major US cities. Regional carriers like U.S. Cellular, Consumer Cellular, Mint, Cricket, and more are also seeing outages. However, some of those are MVNOs which means that if the major carrier they lease from is having trouble, they likely will too.
There’s no word yet from the carriers on what is causing the issues. Here’s the map view of reported outages from Downdetector for the major carriers:
@TMobileHelp is there a service outage? I cant text or call but I can use the internet to send this tweet lol
— Cameron Hegger (@CameronHegger) June 15, 2020
https://twitter.com/privatepardi/status/1272619745255329792
https://twitter.com/itsevejf/status/1272610660808101888
Update: President of Technology at T-Mobile, Neville Ray apologized for the nationwide outage on Twitter and simply said “Our engineers are working to resolve a voice and data issue that has been affecting customers around the country.” As for a timeline on getting the outage fixed, he said they “hope to have this fixed shortly.”
Meanwhile, AT&T told The Verge that its network is “operating normally.” The Verge suggests that the major T-Mobile outage could be causing users of other carriers to think their service is down (due to failed calls) when it could be just on T-Mobile’s end.
Our engineers are working to resolve a voice and data issue that has been affecting customers around the country. We’re sorry for the inconvenience and hope to have this fixed shortly.
— Neville Ray (@NevilleRay) June 15, 2020
Signs are pointing to the big T-Mobile outage being the problem with failed calls to T-Mobile users causing confusion. Verizon gave a general statement hinting at that to The Verge.
“We’re aware that another carrier is having network issues. Calls to and from that carrier may receive an error message,” reads part of a statement from Verizon.
While work continues to restore T-Mobile service, the company’s support team has suggested customers use FaceTime and other third-party calling services.
Our engineers are working to resolve a widespread routing issue affecting voice & text. Customers may experience longer care wait times. Please try third-party calling apps (FaceTime, WhatsApp, Signal) as a temp solution.
— T-Mobile Help (@TMobileHelp) June 15, 2020
Teams continue to work as quickly as possible to fix the voice & messaging problems some are seeing.
Data services are now available & some calls are completing. Alternate services like WhatsApp, Signal, iMessage, Facetime etc. are available. Thanks for your patience. https://t.co/uQiGSAFEAH— Neville Ray (@NevilleRay) June 15, 2020
Update 6:55 p.m. PT: Hours later and T-Mobile is still working on the challenging problem. If you’ve been hearing about the issues today stemming from a huge DDoS attack, here’s a great debunk thread from Cloudflare founder, Matthew Prince:
It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks. 2/X
— Matthew Prince 🌥 (@eastdakota) June 15, 2020
That caused a lot of T-Mobile users to complain on Twitter and other forums that they weren’t able to reach popular services. Then services like Down Detector scraps Twitter and report those services as being offline. 3/X pic.twitter.com/SdscxRaCcZ
— Matthew Prince 🌥 (@eastdakota) June 15, 2020
So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services so that’s not surprising. 4/X
— Matthew Prince 🌥 (@eastdakota) June 16, 2020
From @Cloudflare’s vantage point, we can see a number of things that show there is no massive DDoS attack. First, traffic from WARP to supposedly impacted services is normal and has no increase in errors. 5/X pic.twitter.com/vGPcWAaRgn
— Matthew Prince 🌥 (@eastdakota) June 16, 2020
Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks and definitely would during one allegedly this disruptive. 6/X pic.twitter.com/5zZCRwjsWo
— Matthew Prince 🌥 (@eastdakota) June 16, 2020
Finally, our team know the network operators at nearly all the other major Internet services and platforms and none of them are reporting anything anomalous. 7/X
— Matthew Prince 🌥 (@eastdakota) June 16, 2020
Except T-Mobile, which is having a bad day almost certainly entirely of their own team’s making. So, please, #hugops. And don’t worry, this is one thing that does not need to get added to the list of craziness that has been 2020. 8/8 pic.twitter.com/VONBSSlhLG
— Matthew Prince 🌥 (@eastdakota) June 16, 2020
After a more than 12-hour outage, T-Mobile has solved the major outage:
We are seeing recovery for those impacted by voice and text issues. For full update please read: https://t.co/sDXZemXRsK
— Neville Ray (@NevilleRay) June 16, 2020
FTC: We use income earning auto affiliate links. More.
Comments