The FCC’s Public Safety and Homeland Security Bureau (Bureau) has released a report containing its findings and recommendations on the March 8, 2017 AT&T VoLTE 911 outage. On the afternoon of March 8, nearly all AT&T VoLTE customers nationwide lost 911 service for five hours, resulting in failed 911 call attempts from an estimated 12,600 users. The Bureau’s report finds that the failures triggering the outage occurred entirely within AT&T’s network. Specifically, AT&T placed an incorrect record of whitelisted IP addresses into its customer provisioning system and initiated a network change that pushed the record containing the incorrect whitelist onto AT&T’s live network. This change inadvertently broke AT&T’s connection with Comtech, one of its 911 call routing subcontractors, and thereby disrupted the flow of information regarding the appropriate PSAP to receive certain 911 calls to AT&T’s network. The Bureau’s report notes that AT&T has taken steps to address the vulnerabilities that caused the outage, and that the Bureau anticipates that these changes will help prevent the recurrence of a similar 911 outage and help AT&T with future outage detection and remediation. The report also states that had AT&T followed CSRIC’s recommended network reliability best practices, it could have prevented the outage or mitigated its impact. To protect against similar outages in the future, the Bureau plans to engage in stakeholder outreach and guidance regarding CSRIC’s recommended best practices.
