(CNN) — AT&T’s network went down for many of its customers across the United States Thursday morning, leaving customers unable to place calls, text or access the internet.
By a little after 3 pm ET, roughly 11 hours after reports of the outage first emerged, the company said that it had restored service to all impacted customers.
“We have restored wireless service to all our affected customers. We sincerely apologize to them,” AT&T said in a statement. The company added that it is “taking steps to ensure our customers do not experience this again in the future.”
Later on Thursday, AT&T said an “initial review” of the outage found it may have been caused by an error within the company and not a cyberattack.
The Federal Communications Commission confirmed Thursday afternoon that it is investigating the outage. The White House says federal agencies are in touch with AT&T about network outages but that it doesn’t have all the answers yet on what exactly led to the interruptions.
Although Verizon and T-Mobile customers reported some network outages, too, they appeared far less widespread. T-Mobile and Verizon said their networks were unaffected by AT&T’s service outage and customers reporting outages may have been unable to reach customers who use AT&T.
Thursday morning, more than 74,000 AT&T customers reported outages on digital-service tracking site DownDetector, with service disruptions beginning around 4 am ET. That’s not a comprehensive number: It tracks only self-reported outages. Reports had been rising steadily throughout the morning but leveled off in the 9 am ET hour. By 12:30 pm ET, the DownDetector data showed some 25,000 AT&T customers still reporting outages. By 2 pm ET, fewer than 5,000 customers were still reporting issues.
Earlier Thursday, AT&T acknowledged that it had a widespread outage but did not provide a reason for the system failure.
“Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them,” AT&T said in a statement at 11:15 am ET. “We encourage the use of Wi-Fi calling until service is restored.”
To set up Wi-Fi calling, users can go to their Settings app on their phone. iPhone users should tap “Cellular” and Android users should click “Connection” and then users will be prompted to turn on the Wi-Fi calling feature. AT&T says on its website that there is no extra cost for this feature. Once set up, Wi-Fi calling works automatically when you’re connected to a Wi-Fi network that you choose.
By late morning, AT&T said most of its network was back online, and it confirmed Thursday afternoon that service was fully restored.
AT&T’s stock fell more than 2% Thursday, a an outlier on a day when the market was rocketing higher.
A faulty software update?
AT&T has encountered sporadic outages over the past few days, including a temporary 911 outage in some parts of the southeastern United States. Although outages happen from time to time, nationwide, prolonged outages are exceedingly rare.
On Thursday, the company said it believed the massive outage was caused by an internal issue.
“Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” AT&T said in a statement on its website. “We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
According to an industry source who spoke on the condition of anonymity, the root of the outage appeared to be related to how cellular services hand off calls from one network to the next, a process known as peering.
Though AT&T said the issue does not appear to be a cyber attack, a US cyber official tracking the AT&T outage told CNN on Thursday that they were investigating it.
The US Cybersecurity and Infrastructure Security Agency is “working closely with AT&T to understand the cause of the outage and its impacts, and stand[s] ready to offer any assistance needed,” Eric Goldstein, the agency’s executive assistant director for cybersecurity, said in a statement to CNN.
Carriers are notoriously mum about why their networks go down. In the past, there have been construction accidents that have cut fiberoptic cables, incidents of sabotage or network updates filled with bugs that became difficult to roll back.
Outages often happen for mundane reasons, several telecom experts told CNN.
Common causes include construction-related digging that punctures fiber optic cables and software misconfigurations that can lead to interruptions, said TJ Kennedy, a public safety communications expert.
“I can’t think of every incident in the last few years, but I can think of things related to routers, things related to backhaul, things related to software,” Kennedy said. “This has happened across all major carriers, multiple times in the past few years alone.”
Thursday’s outage could have been caused by human errors in AT&T’s cloud-based networking system, said Lee McKnight, an associate professor at the Syracuse University School of Information Studies.
“The dirty secret of telecom networks these days is they are just a bunch of wires and towers connected to the cloud,” McKnight said. “Someone making a mistake, and others on their team — and their automated tools — not catching it, is quite common in cloud computing.”
The FCC is investigating
The Federal Communications Commission confirmed Thursday afternoon that it is investigating the incident.
“We are aware of the reported wireless outages, and our Public Safety and Homeland Security Bureau is actively investigating,” the FCC said in a statement Thursday afternoon that was posted on X. “We are in touch with AT&T and public safety authorities, including FirstNet, as well as other providers”
The FCC requires carriers to report information linked to network disruptions.
Ahead of the news that the FCC was probing the outage, a former FCC official told CNN: “The carriers are required to report their outage numbers over time, and the commission can track the number of consumers and cell sites down and things like that.”
Fines may be possible in connection with 911 outages, although they aren’t a certainty, said Blair Levin, a telecom policy analyst and another former FCC official.
“The FCC cares a lot more about the inability to connect with 911 [than other types of calls],” said Levin. “It’s a more serious problem from the FCC’s perspective.”
Telecom carriers have every reason to fix any outages quickly, said the first former FCC official, “because it creates black eyes for the brand.”
“Everybody’s incentives are aligned,” the former official said. “The FCC is going to want to know what caused it so that lessons can be learned. And if they find malfeasance or bad actions or, just poor quality of oversight of the network, they have the latitude to act.”
Local governments report outages
Several local governments said AT&T’s outage was disrupting their services.
San Francisco’s Department of Emergency Management said in a statement on X Thursday morning that its 911 center remained operational, but many AT&T customers were unable to reach the emergency line because of the outage. It suggested people call from a landline or find someone with a rival’s service to dial 911.
“We are aware of an issue impacting AT&T wireless customers from making and receiving any phone calls (including to 911),” the department said in its post. “We are actively engaged and monitoring this.”
The Fire Department in Upper Arlington, Ohio, said the AT&T outage was affecting its fire alarms. St. Joseph County, Michigan, advised residents to use Wi-Fi to place 911 calls if they can’t reach 911 on AT&T’s network. Cobb County, Georgia, said its 911 operations remained unaffected by the outage but noted customers may want to find alternate methods of reaching emergency services. Cabel County, West Virginia, said customers that couldn’t reach 911 could text to 911 as a last resort.
New York Police Department officials told CNN that they were not able to make calls or utilize emails on AT&T phones Thursday morning unless they were connected to Wi-Fi.
The Massachusetts State Police warned people not to test their phone service by placing 911 calls.
“Many 911 centers in the state are getting flooded w/ calls from people trying to see if 911 works from their cell phone. Please do not do this,” the state police said in a post on X. “If you can successfully place a non-emergency call to another number via your cell service then your 911 service will also work.”
Atlanta Mayor Andre Dickens, meanwhile, said the city is “actively gathering information to determine how the City of Atlanta can assist in resolving this issue,” in a statement posted on X. Dickens said Atlanta’s “e-911 is able to receive inbound and make outbound calls” and encouraged AT&T customers to direct inquiries to restore service to the company.
An AT&T spokesman said the company’s FirstNet network has remained operational. FirstNet provides coverage for first responders and is advertised as a more robust network than the AT&T commercial network. It uses a mix of its own infrastructure plus AT&T’s broader network. Its customers include police and fire departments, as well as first responders during natural disasters.
Verizon and T-Mobile say they’re unaffected
There also have been about 1,000 outages reported by both Verizon and T-Mobile customers Thursday morning, the DownDetector website indicates.
“We did not experience an outage,” T-Mobile said in a statement. “Our network is operating normally.”
Verizon had a similar comment, saying it was unaffected by AT&T’s outage.
“Verizon’s network is operating normally,” Verizon told CNN in a statement. “Some customers experienced issues this morning when calling or texting with customers served by another carrier. We are continuing to monitor the situation.”
User reports on Downdetector about a T-Mobile outage, the company added, are “likely reflecting challenges our customers were having attempting to connect to users on other networks.”
Downdetector offers “real-time status information for over 12,000 services across 47 websites representing 47 countries,” the website says.
This story has been updated with additional developments and context.
CNN’s Caroll Alvarado, Sean Lyngaas, John Miller, Kevin Liptak and Samantha Delouya contributed to this report.
The-CNN-Wire
™ & © 2024 Cable News Network, Inc., a Warner Bros. Discovery Company. All rights reserved.
