New York Times Headline: Delta Malfunction on Land Keeps a Fleet of Planes From the Sky

Voicing that opinion in combination with the related incidents for many years, this is just another example that we rely solely on cloud technologies without proper (working) backup or fallback systems.

Talking about “system failure”, it might not be related to technology here…

No matter if it was a power failure as initially stated or a malfunction of a power control module, it shows that Delta IT did not do basic precautionary homework. No backup power, no working (tested) system redundancy at a different server location. So one server location (Atlanta) fails and down goes Delta…? I can understand if a home location fails on backup and redundancy, but no serious company should rely on a single location. Redundant Array of Independent Disks (RAID) is a common precaution and on distributed systems can also operate at geographically different locations in order to assure that during an outage at one location, the other location seamlessly takes over.

The secondary system may be slower, less responsive, but it provides the backup.

And Delta cannot claim that to be the first time. The first time I had a similar case was with Northwest (now Delta) back in early 1990, where Northwest was one of the only flights leaving Frankfurt on time during a system outage, as we issued boarding passes by hand, based on the passenger list printed the evening before… I think that was the last time no individually traveling passenger had to sit on the middle seat alone…

Another example I keep referring to was the time in the later 90s, early 2000s during the infancy of online travel booking, when Lufthansa hat to shut down Expedia. As Expedia inquiries for flight availability paralyzed Lufthansa operations. Because from the old logic of the airline CRS (computer logic), the booking process prioritized operational processes. Expedia’s uncached availability requests to the Lufthansa hosts flooded the Lufthansa system to the point where no operational requests could be processed. Good night check-in, good night passenger manifests, good night operations.

In 2004, a system crash paralyzed Lufthansa, which I addressed in my blog about the St. Florian’s Principle: Oh Holy Dear St. Florian, don’t burn my house, take the neighbor’s one.

Those are just major ones, which became noticeable to me. Business Insider reports more cases with JetBlue, American and United. So now it’s Delta. But taken the speed of IT development and the increasing complexity of the systems used, I doubt it takes a long time to hear  about more such crashes. Time for the airline IT to do it’s homework and make sure the host system (CRS) is redundant…

Food For Thought
Comments welcome