Engineer’s password problem contributed to air traffic control failure – report

Resolving an air traffic control (ATC) meltdown in August 2023 was made more difficult because of delays in verifying the password of an engineer allowed to work remotely, an inquiry has found.

More than 700,000 passengers suffered disruption when flights were grounded at UK airports on August 28 last year after ATC provider National Air Traffic Services (Nats) suffered a technical glitch while processing a flight plan.

An inquiry set up by regulator the Civil Aviation Authority (CAA) found that Nats rostered a Level 2 engineer to be on call rather than on site that day, despite it being one of the busiest of the year in terms of flight passenger numbers.

A more junior Level 1 engineer, who was on site at Nats’ headquarters in Swanwick, Hampshire, began checks as soon as automatic flight planning systems failed at 8.32am.

The Level 2 engineer was contacted 34 minutes later but their password login details “could not be readily verified due to the architecture of the system”, the report stated.

After exhausting remote intervention options, it was agreed they would attend the control centre but it took a further one-and-a-half hours for them to arrive, which was three hours and 15 minutes after the incident began.

Nats should consider rostering a Level 2 engineer on site during busy periods such as the summer, the inquiry found.

It acknowledged this would be a “significant” expense, but insisted it should be viewed in “the context” of the overall cost to the industry and passengers from the August 28 2023 failure, which it estimated at reaching up to £100 million.

Our report sets out a number of recommendations aimed at improving Nats' operations and, even more importantly, ways in which the aviation sector as a whole should work together more closely to ensure that, if something like this does ever happen again, passengers are better looked after

The report also noted that the assistance of Frequentis Comsoft, which manufactured the system that shut down, was “not sought for more than four hours after the initial event”.

A solution was identified by the company within 30 minutes of it being contacted.

The inquiry was led by Jeff Halliwell, who has served as a chief executive and non-executive director in roles across the private and public sector.

He said: “Our report sets out a number of recommendations aimed at improving Nats’ operations and, even more importantly, ways in which the aviation sector as a whole should work together more closely to ensure that, if something like this does ever happen again, passengers are better looked after.”

The report recommended that Nats should give earlier notice to airlines and airports of possible disruption.

Some aviation organisations said it took too long for them to be informed about the August 2023 failure, with several first hearing of it from media coverage.

Eurocontrol, which coordinates ATC in Europe, was not alerted until 10.43am, more than two hours after the incident began.

The report said: “Most of the airlines and airport representatives agreed very strongly that earlier warning of a potential problem would have made a considerable difference to their ability to make precautionary preparations, which in turn would have reduced the negative impact on passengers.”

There were also concerns over the “quality and style” of Nats’ communications, and “considerable frustration about the inability to ask questions or to find out detailed information”, it added.

Airlines and passengers were severely let down ... An ATC failure of this scale, a crucial part of national infrastructure, can never be allowed to happen again

EasyJet chief executive Johan Lundgren said airlines and passengers were “severely let down by Nats”, and a failure of this scale “can never be allowed to happen again”.

Ryanair chief executive Michael O’Leary called on Transport Secretary Louise Haigh to “take immediate action to fix Nats’ hopeless service”, starting by sacking its “overpaid and underperforming” chief executive Martin Rolfe.

The inquiry found that an automatic flight planning system and its back-up shut down within 20 seconds after a plan for a flight from Los Angeles to Paris (Orly) was received.

This was because of a “unique set of circumstances not previously encountered”, including a pair of duplicate three-letter waypoints, which are used to identify locations.

The system had previously processed more than 15 million flight plans without this scenario being seen.

The failure led to flight plans being manually processed, decreasing the rate from up to 800 per hour to 60 per hour.

Restrictions on the number of flights allowed to take off from UK airports were in place from 11am until 6.03pm on August 28, but disruption continued for several days due to aircraft and crew being in the wrong location.

We fixed the specific issue that caused the problem last year as our first priority and it cannot reoccur

A Nats spokesman apologised for the “inconvenience passengers suffered” because of the “very unusual technical incident”.

He went on: “Our own internal investigation made 48 recommendations, most of which we have already implemented; these include improving our engagement with our airline and airport customers, our wider contingency and crisis response, and our engineering support processes.

“We fixed the specific issue that caused the problem last year as our first priority and it cannot reoccur.

“We will study the independent review report very carefully for any recommendations we have not already addressed and will support their industry-wide recommendations.”

The inquiry noted that a number of affected passengers waited “many weeks, and in some cases months” for airlines to refund their out-of-pocket expenses.

It recommended that the CAA is given the power to “take consumer enforcement action” without going through the courts, which could include the ability to fine airlines.

The Nats IT failure last year was an unprecedented event that we all hope never happens again

Ms Haigh said: “My priority is to ensure all passengers feel confident when they fly.

“That’s why my department will look to introduce reforms, when we can, to provide air travellers with the highest level of protection possible.”