By Ian Duncan, Michael Laris, Lori Aratani
The Federal Aviation Administration said that a preliminary analysis of a Wednesday meltdown that grounded the US’ aeroplanes showed a data file was damaged by personnel who failed to follow procedures.
As the nation's airspace recovered Thursday, attention turned to the complicated patchwork of old technology and modern additions that make up the ageing safety system. The FAA revoked access to the system for the personnel involved.
The FAA system, which distributes what is known as Notice to Air Missions (NOTAM) safety updates, has parts up to 30 years old, according to a person familiar with the problems, underscoring how the national aviation system relies on ageing technology.
The agency has been working on upgrading the system, the person said, but those older parts and newer ones remain dependent on each other.
The latest problem came after months of disruptions to air travel and revealed a new dimension to the fragility of the aviation system.
The FAA system failure also raises the stakes for a major legislative package before Congress in the coming months that will set the agency's direction for years. Key lawmakers have vowed to investigate what went wrong as they turn to writing the legislation.
Acting FAA administrator Billy Nolen cancelled a planned trip to Mexico on Thursday as the agency investigated the cause of the problem, which the FAA blamed on a damaged database file.
As the failure unfolded on Wednesday, the top FAA official responsible for the air traffic system was handling the issue remotely as he attended an industry conference in Hawaii.
The agency decided to reboot the system and issue a nationwide ground stop Wednesday morning, the first of its kind since the 9/11 attacks, bringing commercial air travel to a standstill.
The ground stop was lifted after about 90 minutes, but delays continued through the day as airlines sought to manage the ensuing crush of traffic.
By the end of the day Wednesday, almost 11 000 flights had been delayed, and an additional 1 353 were cancelled, according to tracking service FlightAware. On Thursday, 4 600 flights were delayed, and 160 were cancelled.
The FAA's computer woes followed a week-long meltdown at Southwest Airlines, roughly between Christmas and New Year's, that executives and union leaders linked to ageing computer systems.
The issues also came on the heels of growing dissatisfaction with air travel after a summer marred by elevated numbers of delays and cancellations.
While many previous issues largely stemmed from internal problems at airlines, the latest disruption put the federal government's infrastructure in the spotlight. Transportation Secretary Pete Buttigieg pledged on Wednesday to find the root cause of the problem.
"With a government system, we're going to own it, we're going to find it, and we're going to fix it," he said.
Another key issue under examination is why the FAA's backup NOTAM databases in Oklahoma and New Jersey did not prevent the nationwide disruption.
The NOTAM system, little known to the public, but is a way to share safety updates with pilots and underpins the inner workings of the aviation system. Pilots are supposed to consult the bulletins - which can provide warnings of closed runways and other hazards - before departing.
"From a pilot perspective, NOTAMs can keep you alive, they can keep you from getting hurt," said Dennis Tajer, an American Airlines pilot and spokesperson for the Allied Pilots Association. "It's the safety net to ensure the day-to-day things that can happen at airports don't end up with people getting hurt or killed."
The current set-up is a mix of an older system, known as the US NOTAM system and a newer one, dubbed the Federal NOTAM system. The person familiar with the system, who spoke on the condition of anonymity to discuss the probe, described it as a "Frankenstein set-up."
Both systems must constantly sync with each other to share information, leaving a complex hybrid tying together legacy equipment and more-modern parts.
Wednesday's meltdown also brought renewed attention to years-long efforts to upgrade the system, which has faced criticism as being cluttered with irrelevant information that could obscure the safety alerts it was meant to provide.
The FAA said it has consolidated NOTAM information into one location for ease of use and facilitated access for computers to ingest data, and is working on other improvements.
Jose Alfonso, who is the FAA's director of Aeronautical Information Services and has helped oversee changes of the NOTAM system, said in a 2021 FAA briefing that a key challenge is making sure the diverse groups of people who use the national airspace - including those who fly commercial airlines, low-flying agricultural planes and helicopters - receive notices that pertain to them.
"Trust me - we've heard our stakeholders loud and clear on that, and we are doing our best to come up with a way to keep that information that's relevant to the stakeholder up front," Alfonso said in the briefing.
He compared the ongoing effort to provide an "optimal NOTAM" to the 1960s TV series "Dragnet" and Sgt Joe Friday: "Just give me the facts, and nothing but the facts," Alfonso said.
Mike Santoro, a Southwest Airlines pilot and vice president of the Southwest Airlines Pilots Association, said the NOTAM system could be cumbersome, with pages of notices written in jargon. After more than 20 years of flying, he's developed a system for paging through NOTAMs to find reports that are most relevant.
"They could definitely revamp the system to make it more user-friendly," he said. "There are just things in there that don't need to be there."
A July 2017 incident showed the danger of a cluttered system. When an Air Canada flight tried to land in San Francisco that month, the plane nearly crashed into four other airliners.
The runway the Air Canada pilots were looking for had been closed, but the information was buried in a NOTAM, according to National Transportation Safety Board investigators.
The NTSB found in the course of its investigation that the notices often were ignored by pilots because they were unintelligible.
The failure of the system is likely to raise fresh questions about the FAA's ability to manage major technology projects and modernize its systems. Republican lawmakers said the problems show the need for new leadership at the agency, which has been without a Senate-confirmed administrator since last spring.
New details also emerged Thursday of how the outage unfolded Tuesday afternoon.
Santoro said he first learned of problems with the system during the union's regular monthly meeting with leaders of Southwest's flight operations division at about 5pm Tuesday.
"They were just starting to have problems with the system, and they were trying to figure out how to work around those," he said. "There were some glitches, but the system hadn't crashed."
He said pilots could work around the system by calling individual airports to get the latest notices, but at a large airport such as Atlanta's Hartsfield-Jackson, the volume of calls would quickly overwhelm personnel.
American Airlines chief executive Robert Isom said airline leaders on Wednesday discussed with the FAA's Nolen the safest way to manage the outage.
"I'm really pleased with the FAA in terms of calling a timeout yesterday," Isom said in a Thursday interview on CNBC's "Squawk Box." "We talked about, 'Hey, what's best to do?' Safety first."
Isom said investment in the FAA is necessary, likely to the tune of billions of dollars, for projects that would need to be planned over several years.
As FAA officials and airline leaders were scrambling to respond, many top industry and government officials were in Maui for an airports conference. Among them was Tim Arel, the FAA's chief operating officer for the air traffic organisation, who the agency said was fully involved in the response.
Dave Spero, president of Professional Aviation Safety Specialists, an FAA union, said Arel apologised to attendees for having to step away as he worked to restore the failing system, adding that "he was on it all day."
The conference was focused on the aviation industry's agenda this year in Washington, where the FAA legislative package will allow Congress to set new priorities for the agency.
Spero said that a government funding bill contained a provision that should help the FAA better plan its workforce needs and that the next round of legislation is a chance to put the agency on a better path.
"This is an example of how the FAA resources right now are spread so thin that they're unable to prioritize things that need to get taken care of," he said.
Sara Nelson, president of the Association of Flight Attendants union, said the FAA needs certainty going into the future.
"We'll find out more about the root cause of the issue in the coming days, but what's clear is the need for robust and stable funding this year to bring our aviation system into the 21st century," she said in a statement.
Some safety experts cautioned against reaching broader conclusions before a fact-finding process is complete.
"I don't know that one event like this says anything about the longer-term reliability" of the NOTAM system, said Mark Millam, a technical expert at the Flight Safety Foundation and a former airline safety chief.
"Just because there's some loophole of some kind that's detected, it doesn't mean the overall system reliability is not at a functioning and acceptable level."