Living in Jerusalem, in a highly international community, there’s nothing unusual about having a friend traveling out of the country. I’m used to seeing Facebook posts with pictures of one of my neighbors on a beach in Thailand or in some small African village, or back in the US visiting parents. The destinations are periodically a big deal, but except for the early COVID days, we were all boarding airplanes so often that the flights themselves were not. Perhaps that’s why it stuck in my mind when my friend Jonathan posted a picture of himself standing in front of the plane he was about to board on March 10, 2019, an Ethiopian Air flight from Addis Ababa, Ethiopia to Nairobi, Kenya on his way to the United Nations Environment Assembly. I couldn’t help wondering what compelled him to post a picture of himself in front of the airplane and share his flight details?
I only wondered at this for a second before scrolling on. After all, unusual posts are not that unusual. I didn’t think of it again until a few hours later when I heard that Ethiopian Air flight ET302 from Addis Ababa to Nairobi had crashed.
In early 1994, our newly formed Colorado College Chess Club played a match against a local Colorado Springs club. Our coach arranged the evening despite knowing we were seriously outmatched, and in doing so, employed many of the Hurry Up and Fail principles we’ve already discussed. He wanted us to start before we were ready and pit ourselves against far superior opponents so we could see how much more we had to learn.
Predictably, we got creamed. Every single one of my teammates lost. In my match, I was playing black and was doing OK. My opponent used an opening I’d never seen before, but I countered it as best as I could, and after 45 minutes, the pieces were still even. Our club coach said he wanted to talk to me privately. My opponent immediately objected.
Our coach was a chess master. When recruiting for the club, he arranged tables around himself in a rectangle configuration, set up 12 chess boards, and invited all challengers. Over the next hour, he easily defeated over 50 students, playing a dozen games at a time. There was no doubt he could destroy my current opponent. But this wasn’t a football game where the coach could call a time out and have a one-on-one with the quarterback. Chess isn’t a team sport, and it seemed wrong for him to jump in and coach me on strategy.
My coach assured my opponent and me that this was perfectly normal, and he wouldn’t give me any moves. What he said instead was simple, yet somehow I missed it. He said I should offer my opponent a draw. If he accepted, we’d still leave the evening winless, but at least avoid losing every game. Unfortunately, I misunderstood, thinking he wanted me to play for a draw, something I’d no idea how to do.
The interchange left me flustered and confused. My defense quickly fell apart, and I lost within minutes.
But my big revelation for the night was still to come. During our game, my opponent and I recorded our moves. After defeating me and shaking my hand, my opponent set up the board anew. He then used our recordings to review the game step by step. He explained the opening he used and why he liked it. He even said my impromptu defense used his preferred counter against that attack. Of course, my off-the-cuff moves didn’t follow his preferred defense for long. He asked me what I was trying to achieve with my tactics and showed me alternatives that might have worked better.
Then we got to the disastrous turning point where I abandoned my strategy and tried to play for a draw. He showed me the end game tactics that might have worked better, though he also explained why, despite having the same pieces, his position was already better and would have led to victory even if my defense hadn’t collapsed.
This process is known in chess circles as a post-mortem. Post-mortem is Latin for “after death.” Medically, it’s another term for an autopsy, an examination of a corpse to determine the time and cause of death. In chess, the post-mortem is a primary teaching tool. My opponent during the match became my teacher after it. We reexamined my moves, identifying where I went wrong and how I could do better.
During that first post-mortem, I mostly listened and asked questions, lacking the knowledge to help my opponent improve his own game. But from that point on, the post-mortem became a regular tool in my toolbox, both for chess and beyond.
In the early 1950s, the world was just entering the age of jet-powered commercial flight. Jet planes flew faster, higher, and quieter than their propeller-powered predecessors. But the new technology brought new challenges. Crashes were not uncommon, and the wreckage left few clues as to what went wrong.
In 1953, following multiple mysterious crashes of the de Havilland Comet airliners, Australian David Warren decided to do something about it. He realized our ability to make planes safer was severely inhibited by our failure to understand why they crashed. Post-mortem investigations suffered from a lack of data. His idea was simple, even if the technology behind it was not. He wanted a way of recording what happened in a plane before a crash that could be recovered post-crash so an investigative team could accurately assess its cause.
His solution was to create a Flight Data Recorder (FDR) so solid it could survive a plane crash. Early FDRs contained up to four hours of audio recordings as well as data including the plane’s heading, altitude, airspeed, vertical accelerations, and the time. Warren initially received pushback as detractors claimed recording cockpit conversations would violate a pilot’s right to privacy. But as jetliners continued to fall out of the sky, privacy issues became secondary. Before long, Warren’s FDRs were mandated on virtually all commercial flights. Though colored a bright orange to make them easy to spot amongst the wreckage, the FDRs are colloquially known as “black boxes,” and have been instrumental in advancing airline safety.
I started this chapter with a story about my friend, Jonathan, the one traveling Ethiopian Air. He arrived safely in Nairobi blissfully unaware friends and family around the world were in a panic and thought they’d lost him. He learned upon landing that a different Ethiopian flight from Addis Ababa to Nairobi, the one that had taken off directly before his, hadn’t made it. All 157 passengers and crew died, including over a dozen who were also traveling to the United Nations Environment Assembly meeting.
The plane was brand new, the pilot well-trained and experienced, the weather was clear. So what had gone wrong?
The history of air travel contains more than its share of freak accidents. But many suspected this crash was more than an anomaly. Why? Because the circumstances of the Ethiopian disaster were eerily similar to a crash less than a year earlier on Lion Air Flight JT610 from Jakarta to Pangkal Pinang. Both flights crashed within minutes of taking off. And both flights were on identical equipment, the newly released Boeing 737 Max.
Ethiopian Airlines Flight ET302 wasn’t one of those crash landings where the pilot glides the plane down and lands it as softly as possible. This plane dove at a 40-degree angle, hitting the ground at over 500 miles per hour, hard enough to leave a crater 32 feet deep, and so utterly destroying the aircraft that few pieces of the wreckage remained large enough to tell they once belonged to a plane.
From 2010 to 2019, approximately 57% percent of all airline crashes were attributed to pilot error, 21% to mechanical problems, and 10% to adverse weather conditions. In actuality, few of these crashes involved just one factor. For instance, a pilot may make a poor choice when flying into strong winds, but do we conclude the crash was due to poor weather or to pilot error? As safety expert Sidney Dekker says, human error is not an explanation. Human error demands explanation.
No investigation has rocked the aviation industry in recent memory more than that of the Boeing 737 Max aircraft. When the black box data from flights ET302 and JT610 were analyzed, they didn’t reveal anything as innocuous as a pilot error or poor weather. They revealed a defect in the entire industry.
If you read the crash reports, you’ll discover two flaws that in combination brought down both flights. The first was a faulty sensor, which relayed incorrect data to the computer. The second was an onboard computer system called the Maneuvering Characteristics Augmentation System (MCAS) which tried to stabilize the airplanes. But since the MCAS was relying on faulty data, it doomed the planes instead. This might be humanity’s second biggest fear with artificial intelligence. When computers have so much power over our lives, one faulty instruction or incorrect data point can lead to disaster. (For humanity’s number one fear with AI, see Terminator, The)
When the pilots discovered their planes were trying to kill them, couldn’t they have turned off the MCAS? In theory, yes. However, both crashes occurred soon after takeoff. There wasn’t much time to react. More importantly, the Ethiopian pilots weren’t well trained on how to deal with the MCAS, and the Lion Air pilots didn’t know it was there at all. Before you conclude this resulted from poor training from third world airlines, it wasn’t. Boeing intentionally hid its very existence.
It seems counterintuitive that Boeing wouldn’t want pilots trained on how to recover from an MCAS error. But like airline crashes, horrible business decisions are often not the result of one factor alone. Boeing felt pressure from at least two different parties, a competitor and a client.
Airbus and Boeing are the world’s two largest airplane manufacturers by a wide margin. They’re in constant competition for contracts from major airlines, and when either gains a technological advantage, the other feels tremendous pressure to match it.
On December 1, 2010, Airbus released the A320neo. As a passenger, when I think about airplane improvements, I care most about the comfort of seats, legroom, and entertainment options. But to airlines, there’s a far bigger concern: fuel efficiency. For perspective, according to the Southwest Airlines 2012 annual report, fuel costs reached 6.156 billion dollars, making up 37.3% of their annual operating expenses.
The A320neo shocked the airline industry by offering 15% to 20% better fuel efficiency on a per-seat basis than the prior A320 model. It also promised to be at least 6% more efficient than Boeing’s airplane in the same class, the 737. Let’s apply these savings to Southwest, who posted profits of $417 million in 2012. If Southwest cut just 6% of its 6.156 billion dollar fuel bill, they would have saved over $369 million, increasing their $417 million profit to $786 million, a spike in profitability of over 88%. This wasn’t a small development; it was a game changer. Boeing could do nothing but watch as the Airbus A320neo became the fastest selling aircraft in history, dealing more in a single week at the 2011 Paris Air Show than Boeing had sold 737s in all of 2010.
Boeing needed a new plane to compete with the A320neo, and they needed it fast. Designing a whole new airliner no doubt would have given the company its best chance at technologically outperforming the A320neo. But that would take years and cost billions. With their market share slipping fast, they couldn’t afford delays. Instead, they decided to quickly revamp the 46-year-old 737.
Unfortunately for Boeing, there was a key distinction between the Airbus A320 and their own 737 that made the A320 easier to revamp. The A320 rides higher when it’s on the ground, which left room to fit a larger, more fuel-efficient breed of engine under the wing. Boeing couldn’t easily make the same improvement because of the lack of space under the 737’s wings.
Time was of the essence. To compete in the marketplace, Boeing needed to produce new engines that could fit on their old, low-riding plane. They came up with a solution, albeit an imperfect one. To mount the larger engines on the wings and still have enough clearance below the engines, they made the top of the engines stick up over the top of the wings. This caused the airplanes to have too much lift when in a steep climb. Too much lift can increase what’s called the Angle of Attack, the angle at which the plane diverges from the flow of the surrounding air. If the Angle of Attack exceeds 18 to 20 degrees, a plane is in serious danger of stalling.
There were quite a few potential solutions to this problem, but another restraint limited Boeing’s options.
Let’s go back to Southwest for a moment. If you’ve ever flown Southwest, you know they operate far differently than the old-school airlines like United, American, and Delta. Southwest changed the boarding process and seat assignments, and they eliminated first class and meal service. Why? Because Southwest is an efficiency machine.
In 2019, Southwest recorded its 47th straight year of profitability, something unheard of in the traditionally anemic airline industry. I keep waiting for more airlines to copy Southwest’s playbook, because they’re clearly doing something right; but strangely, they’ve remained an outlier. Their obsession with efficiency goes way beyond innovative boarding and seating plans. One of their biggest distinctions is that they only fly one model of plane: the Boeing 737.
How does keeping their entire fleet flying 737s help? It means all key components are interchangeable. If one plane has mechanical issues, simply bring in another. You don’t have to worry about some flight crews being trained on one piece of equipment and others on different planes. All crews are interchangeable. The mechanics only have to work on one model of plane. You only need to keep spare parts for the 737, which cuts down on inventory and increases the odds of having what you need on hand.
As thrilled as Southwest must have been to learn Boeing was working on an upgrade with better fuel efficiency, they also had one concern. They made it clear to Boeing that they would only buy the 737 Max if it flew exactly like their other 737s. In fact, they negotiated a clause in their purchase contract for the new planes stating Boeing would owe them a rebate of one million dollars per plane if Southwest was required to train their pilots to fly them.
It wasn’t only to satisfy Southwest that Boeing felt pressure to keep the flying experience exactly the same as on the older 737s. Pilots don’t simply shift from flying a 737 one day to a 747 the next. The planes are different, the cockpits are different, the controls are different. Switching from one plane to another poses the risk of pilots getting confused when under stress, and when pilots get confused, people can die. Pilots need to be rated by the FAA for each type of plane they fly, and they’re normally only rated for one type at a time. If Boeing could make the new 737 Max fly exactly like the old 737, they could claim it should be included under the existing FAA 737 rating. Otherwise, the FAA might assign the 737 Max a new rating class, which would limit who could fly them, raise training costs, and undermine orders. In other words, Boeing needed a solution for the stalling problem that would ideally be undetectable to pilots and not require additional training.
Their solution was a secret software override that would kick in when the 737 Max detected too much lift. The MCAS would take control, forcing the nose downwards until the Angle of Attack returned to safe levels. Pilots were neither trained to handle the new software nor even informed it existed.
But what would happen if the sensors measuring the Angle of Attack returned incorrect data? What if the software believed the plane had too much lift, when in actuality, the plane was perfectly safe? In such a scenario, the software would keep pushing the nose of the plane down until the plane was in a steep dive toward the ground below, and its untrained pilot would have no clue how to react.
Was Boeing blissfully unaware of this potential flaw in the 737 Max until the planes starting falling out of the sky? Unfortunately, no. Boeing admits it knew about the problem a year before the Lion Air crash, but still took no action.
A week after the first crash, the pilots finally heard the 737 Max had a “flight control system” that could cause pilots to have “difficulty controlling the airplane” and result in “possible impact with terrain.”
Nevertheless, Boeing was still reluctant to release information about the MCAS or take any steps to correct the problem. Simultaneously, they informed pilots they were the backups if the MCAS system failed. But how could they be the backups for a system they knew nothing about?
Pilots weren’t reassured by either of these statements. As they told Boeing, “We flat out deserve to know what is on our airplanes.”
The FAA didn’t ground the 737 Max planes, but did issue a directive to pilots to turn off the MCAS system if they ever witnessed it trying to send the plane into a steep dive. The Ethiopian Air pilots had received the directive, but no one informed them they’d only have ten seconds to shut the faulty system down. By the time they managed to get it offline, it was too late to recover.
Freak accidents occur in air travel. Black box analysis helps engineers identify errors, and periodic tweaks are made to prevent a recurrence. But when Ethiopian Airlines Flight ET302 went down, the Lion Air crash could no longer be written off as a freak accident. This was now a pattern, and it was a pattern involving Boeing’s newest and best-selling aircraft.
All 387 functioning 737 Max aircraft, which served 8,600 flights per week for 59 airlines, were grounded by March 18, 2019. The groundings continued for 20 months, the longest ever grounding of a US airliner. Boeing was forced to pay an estimated $20 billion in compensation to airlines and victims’ families, lost business, and legal fees, in addition to losing more than $60 billion from 1,200 canceled orders.
The results of the 737 Max black box investigations were damning, not just to Boeing, but to the airline industry as a whole. The FAA admitted it never fully analyzed the MCAS system prior to certifying the 737 Max as fit for flight. In fact, much of the responsibility for certifying the safety of the planes has been shifted in recent years from the FAA to the plane manufacturers themselves. In other words, the regulatory system meant to keep the airline industry honest and passengers safe has been largely outsourced to the very people who have a financial incentive to manipulate the system. The black box analysis didn’t just reveal a technical problem on the 737 Max, it showcased a weakness in our entire regulatory and safety system.
Post-mortem investigations are excellent ways to learn from failures. In this chapter, we looked at post-mortem investigations through two very different lenses, one of them a low-stakes failure, the other a high-stakes disaster.
After losing my chess match, the post-mortem gave me insights into my opponent’s mind, allowing me to understand his strategies and see the flaws in my own. My chess game advanced more in that one post-mortem than it would in a year of casually playing.
The Boeing 737 Max investigation likewise showed the immense power of post-mortems as a tool for learning from failure. However, in this case, there was no Hurrying Up and Failing. Boeing didn’t welcome feedback on their new system, nor did they actively look to expose its flaws so they could correct them. Rather, they attempted to hide known flaws, which meant that when failure did come, it came in catastrophic form.
The next time you’re involved in a failure, don’t just move on and try again. Consider conducting a post-mortem and really examining where things went wrong. The following questions can help you clarify the causes of failure and make better adjustments in the future:
- When did you first detect your effort was likely to fail? Did you immediately take action to get back on the right track or did you avoid facing the failure?
- What was the root cause of your failure?
- Who was involved in the failure? Was there someone working on the project who shouldn’t have been? Was there someone who should have been involved who wasn’t?
- What was the impact of the failure?
- Were any steps taken to mitigate the consequences of the failure? Were there additional steps that could have been taken but weren’t?
- What lessons did you learn from the failure?
- With all this information in hand, how will you proceed differently next time?
The above is just a partial list of questions you can ask as you go through a post-mortem investigation, but I hope they already show how powerful this tool can be in learning from failure.
Of course, the big limitations with post-mortems is that they first require you to fail. If you Hurry Up and Fail, intentionally creating frequent, low-stakes failures, that’s not such a problem. Still, it’s not always easy to engineer low-stakes failures.
Wouldn’t it be great if we could get the benefits of the post-mortem investigation without having to first go through the actual failure? What if we could see impending disaster and avoid it in advance? Just as hindsight is not 20/20, foresight isn’t either. But there is a tool that brings us closer.
