In July 1998, I stood outside Interpol headquarters in Lyon, France, confident in how to blow up the building. Interpol had been bombed once before, in 1986. The explosion caused significant damage but failed to bring down the old headquarters in Paris. The new headquarters in Lyon was far more secure. Or so the organization claimed. But I’d found a vulnerability. Not that I had much power to take action. Fortunately, I wasn’t standing outside Interpol alone, and my companion had all the influence I lacked.
I was nervous to share my strategy as I questioned how responsive my companion would be. So I backed into it with: “Hey Ron, if you wanted to blow up Interpol, could you do it?”
“Easily,” he responded.
“How?”
Ron dove into an explanation of his tactics. His plan lacked subtlety, to say the least, which told me he was unaware of the security vulnerability I’d found. Most importantly, his voice remained light and conversational. That was a good sign. I’d been afraid he’d chew my head off for broaching the subject, but once I heard how he’d proceed with his own strategy, I knew he’d be willing to listen to mine.
Affable as Ron normally came across, he wasn’t one to be trifled with. I’d learned that the hard way when I’d first applied to work with him. My roommate had told me to include a conversation piece on my resume, so I added a blurb about backpacking around the world and included a list of the countries visited.
“I see you’ve been to Cuba,” Ron said after looking over the resume. “Do you know the legal penalty for a US citizen traveling to Cuba?”
This conversation starter was proving a little too effective, and I didn’t like the direction this was going. I knew I’d broken US law when I’d entered Cuba in the winter of 1996, but hadn’t considered it a big deal. I’d never looked into the exact penalty if caught. But Ron knew. And he started telling me the consequences I could have faced. In detail.
Though he was a law professor, I was surprised by his legal knowledge in this area. I doubted anyone else at NYU Law would have had a clue about the issue. “You seem to know an awful lot about the penalties for going to Cuba,” I said.
“Well, that was my branch of government,” Ron replied.
This interview just kept going downhill. “Let me get this straight,” I said. “You used to work for the branch of the government in charge of arresting me when I went to Cuba?”
“No, Dave. I used to run the branch of the government in charge of arresting you when you went to Cuba.”
Google didn’t yet exist, but it didn’t take me long to research the true identity of this mild-mannered law professor. Ronald K. Noble, former Undersecretary of the Treasury for Defense, in charge of the Secret Service, the Customs Service, and the Division of Alcohol, Tobacco, and Firearms (all of which are now under the Department of Homeland Security).
My initial fears that my visit to Cuba would cost me the job proved unfounded. Ron wanted to hire an assistant to help him with a mission that summer at Interpol, and the last thing he wanted was someone he’d have to babysit. He later told me my backpacking experience was key to landing the job. He figured I’d have no problem looking after myself in a foreign country.
And that’s how we came to be standing outside Interpol a few months later, discussing how to blow it up. Once Ron finished telling me how he’d take down the building, I launched into my own strategy, complete with a description of the security vulnerability I’d found.
“No, that can’t be,” he said.
“Check it out yourself,” I replied. And he did. The next day, he got back to me in total shock.
I apologize for not explaining either Ron’s plan or my own. Ron became Interpol Secretary General two years later, so I’m fairly certain he either convinced his predecessor to fix the vulnerability or took care of it himself. But without 100% certainty, I’m not willing to jeopardize Interpol’s security. Besides, the point of this story isn’t to help my readers become more effective terrorists.
So what is the point?
To introduce another powerful tool in the failure toolbox: the pre-mortem.
We already looked at post-mortems, which help us determine the errors that led to a failure so we can do better in the future. Advocates of pre-mortems ask: why delay such an analysis until after a disaster? If you can anticipate danger areas in advance, you can often prevent catastrophe. By asking, “How would I blow up Interpol?” I could discover and repair security vulnerabilities before the organization’s enemies could find them.
Pre-mortems can put the process of learning from failure into hyperspeed as you don’t have to wait for failure to actually occur to learn from it. We saw at the end of the last chapter some of the questions you might ask when doing a post-mortem. The list would look somewhat different with pre-mortems, and might include:
- What are the risks that could potentially bring the project down? Examine if there are any technical issues, resource constraints, scheduling problems or other issues to generate your list of vulnerabilities.
- Which of the risks identified are most likely to lead to failure? Go though each one and identify how much it could contribute to your hypothetical project failure.
- For each risk, can you find a way to a) prevent it from occurring and/or b) reduce the impact if it does occur?
- Who should be in charge of each risk? Failures don’t always come from unexpected sources. Very often, we can see potential sources of failure and still not take action because we assume someone else is handling it. Clearly defining who is accountable for each risk factor can remedy that.
- When should we reexamine our risk factors? Threats can increase or decrease over time. Create regular checkins or milestones ahead of time so these don’t go unchecked.
Potential causes of failure are vast. As we’ll see next, pre-mortem processes can differ greatly depending on whether you’re addressing common or uncommon failure scenarios.
The Sharks on ABC’s Shark Tank have a reputation for driving hard bargains. Many entrepreneurs who’ve pitched their products on the show have been driven to tears by the Sharks’ frank critiques and low valuations. Savvy presenters know how to make a solid, compelling presentation that leaves a little wiggle room for counter-offers, and Cool Jen was nothing if not savvy. She’d only started selling her product a few months earlier and already came into the Tank asking for $100,000 for a 10% share, giving her fledgeling company a million dollar valuation. She immediately got push back from the Sharks. Mr. Wonderful said her offer didn’t give him enough equity to get excited about. Cool Jen told him not to worry, that she was there to negotiate.
Damon Johns started saying her valuation was nuts considering she had almost zero sales, but Mark Cuban, the billionaire owner of the Dallas Mavericks, interrupted Damon to offer Cool Jen $250,000 for a 20% share, raising her company valuation to $1,250,000, but only on condition she took his offer right away. Cool as a cucumber, Cool Jen took the deal in one of the fastest transactions in Shark Tank history.
Cool Jen is better known to the public as Jenny Goldfarb, the owner of Mrs. Goldfarb’s Unreal Deli, a company that produces vegan deli meats. I’m sure she’ll plotz when she reads this segment referring to her as Cool Jen, but almost a decade before she became Mrs. Goldfarb, the nickname Cool Jen had already been well earned in our household through her colorful and overly-hip vocabulary. She’d show up at our door with a “What’s the dizzle?” and greet our newborn with a “What’s up little dude?”
Shark Tank can skyrocket a company’s sales. Each of the Sharks is a highly successful businessperson with a history of profitable product launches. Even products that don’t close deals on the Tank can see sales explode following a pitch, which amounts to a free infomercial before millions of viewers. Cool Jen’s launch was no exception, with her public profile and sales shooting through the roof the moment her episode aired.
But rapid growth is not without its challenges.
Take the case of Mix Bikini, another brand that successfully pitched on Shark Tank. The concept of Mix Bikini was simple. Rather than selling bikinis as sets, Mix Bikini sold tops and bottoms separately, giving customers a wider range of options. Barbara Corcoran, the real estate mogul, liked the idea and invested $50,000 for a 10% share of the company.
The night the Mix Bikini episode of Shark Tank aired, the company held a gigantic party, complete with open bar, swimming pool, and, of course, plenty of bikini-clad women. Barbara herself showed up and partied the night away with her latest investment.
The next day, Mix Bikini’s hung-over CEO dragged himself into the office to see how many sales they made during launch night. He found none. How could millions of people learn about his company and none buy? Could no one have liked the idea?
Just the opposite. Women loved the idea. Too many women. The company’s web server crashed. It hadn’t been strengthened to handle the launch of a major national brand. By the time Mix Bikini brought their website back up, it was too late. They had no way to notify all those would-be customers they were back in business.
A crashed server on launch night turned out to be a disaster scenario for Mix Bikini, which literally never recovered. But the odd thing about the server crash was how predictable it was. Network Administrators regularly run their servers through what’s called a stress test, which simulates the effects of a surge of traffic to see just how much of a load the server can handle.
Those who pitch on Shark Tank aren’t always known for their ability to anticipate problems. Had Mix Bikini not gotten a deal, their failure to prepare for launch night would’ve been understandable. Almost. But entrepreneurs seek Sharks partially for mentorship. Barbara had already been involved in dozens of deals, so it was surprising she didn’t have a team reviewing Mix Bikini’s infrastructure to ensure it could handle the volume.
There’s no rule that says you need to fail yourself to learn from failure. As previously discussed, the pre-mortem can be a great tool for examining a hypothetical failure before it can occur. But the process becomes even easier when dealing with actual failure scenarios that others have already faced. If someone’s previously tried and crashed, it’s a great shortcut to study how and why to prevent the same mistakes. In some areas, you won’t just find evidence of one past disaster. There might be hundreds or even thousands of failures to learn from.
This is the situation the World Health Organization (WHO) found themselves in. Far too many patients were dying from complications during surgery. Even worse, many were entirely avoidable, involving accidents such as performing a surgery on the wrong patient or cutting open the patient’s right side when the issue was on the left. When you have a vast history to learn from, you can pull out the most common causes of failure and include their lessons in your protocols. In the WHO’s case, they reduced complications and fatalities from surgery by over 30% by instituting a simple one-page checklist for surgical teams to go through before each operation. The items on the checklist are all basic, such as a verbal confirmation of the patient’s name, the procedure to be performed, and the location of the procedure. You’d think that information would be so fundamental the checklist would feel like a waste of time, but the thirty-plus percent improvement shows just how often these basic facts were actually missed.
Let’s return to Cool Jen and Mrs. Goldfarb’s Unreal Deli. If you’re hoping for a dramatic climax to the story, I’m sorry to disappoint. Neither Jenny Goldfarb nor Mark Cuban had any intention of allowing Unreal Deli to become another Mix Bikini. Mark’s team contacted Cool Jen and took her through their checklist of preparatory steps for launch night. They recommended having Unreal Deli on Shopify, an ecommerce platform that could handle huge spikes in traffic. Cool Jen already built her site on Shopify, so no adjustment was necessary. In the end, no server crashes or other disasters marred the excitement of Unreal Deli’s launch.
Is this ending anti-climactic? For a book like this one, definitely. It’s anti-climactic in the same way it’s anti-climactic when an airplane arrives on schedule to its designated gate or a patient wakes up after a successful surgery. Flying a plane, going under the knife, and introducing a new product on prime-time TV are high-risk scenarios. While a completely unforeseeable crisis could happen in any of those situations, most of the problems that occur in air travel, surgery, and product launches are predictable. We can easily look back and say the Mix Bikini team should have been less concerned with throwing an epic party and more concerned with their web server strength. Similarly, we can look at a botched surgery where the surgeon operated on the wrong side of the patient and be shocked such errors can still occur.
The takeaway is simple. If you’re in an area with predictable disaster scenarios, learn from past failures and use a checklist to eliminate common errors. Here are factors to take into account to build effective checklists:
- Identify common failure points. Whether its your own failures or you’re identifying industry-wide failures as the WHO did when creating their surgical checklist, you want to find the most common pitfalls.
- Weigh the cost of including an item on your checklist vs leaving it off. The point of the checklist is not to create more bureaucracy. The longer the list, the more inefficiency it can create. It’s worth becoming less efficient to avoid an airplane crash, but if your cost of failure is low, you may opt for a shorter list with only very common problems included.
- Build your list in the correct sequence. The items on the list should be logically organized, usually following the order of operations.
- Keep it simple. The meaning of each item on the checklist should be obvious to everyone. Avoid jargon, acronyms, or technical terms that might confuse team members.
Checklists are fairly straightforward ways of avoiding common problems. But what do you do when the disaster scenarios are less predictable and a basic checklist is insufficient preparation?
On May 12, 2013, Chris Hadfield played David Bowie’s Space Oddity in what Bowie called “possibly the most poignant version of the song ever created.” What made the rendition so powerful? It wasn’t his voice or guitar playing, both of which are fine but hardly at superstar levels. The recording was unique was because it took place 220 miles above the earth’s surface while moving at 17,000 miles per hour aboard the International Space Station (ISS).
Chris’s first voyage into space was aboard the Space Shuttle Atlantis, which was scheduled to dock with the Russian space station Mir on November 15, 1995. A decade prior, such a docking would have been inconceivable. Back then, the United States and the Soviet Union were in the thick of the Cold War, and each one closely guarded their military and space technology. However, in 1991, the Soviet Union collapsed. The US worried advanced Soviet technology could be abandoned, or even worse, fall into unfriendly hands. So the United States drastically shifted policy to cooperate with the Russian Space Agency, even provided funding for important projects.
Before Chris’s mission, a US shuttle had docked with the Mir Space Station once before, but the docking was hacked and awkward. This time, the Atlantis had flown into orbit carrying a five-ton, fifteen-foot-long docking module. Their mission was to permanently affix the docking module to Mir so all future shuttles could easily link up with the Russian station. Docking in space isn’t easy under the best of circumstances. In this case, the Atlantis had a 15-foot tower sticking out of the shuttle blocking their vision and a docking window of only two minutes. Of course, having the tower jutting into space and the short docking window was part of the plan. Having broken sensors was not.
In the previous chapter, we learned about two flights, Ethiopian Airlines Flight ET302 and Lion Air Flight JT610, that crashed when faulty sensors fed their computers incorrect data. Those were just routine flights. This was a US Shuttle and a Russian space station docking together for only the second time in history, anything but routine. As they approached, one of the laser sensors on the shuttle told Chris the shuttle was 32 feet away from Mir. But a second laser sensor told him the shuttle was only 20 feet away. A 12-foot difference is enormous. If the shuttle accidentally attempted to dock 12 feet into the Mir, the two would crash. The Mir would depressurize, killing everyone aboard, and the shuttle could sustain enough damage to burn up upon reentry to the atmosphere.
Fortunately, Chris had been preparing for this moment his entire life. He was 9-years-old when Neil Armstrong walked on the moon, and, like millions of other kids watching, Chris wanted to become an astronaut. At the time, he could see no clear path to ever achieving his goal. The space race between the United States and the Soviets didn’t include Canadians like himself, and it was hard to perceive Canada ever competing on that stage.
Of course, Neil Armstrong also couldn’t have foreseen a realistic path to walking on the moon when he was nine but still did it 29 years later. So the 9-year-old Chris Hadfield decided not to let realism get in the way. Without telling anyone his goal, he made changes to start living like an astronaut and be prepared when opportunity struck. If he woke up tired and sluggish, he’d ask himself, “Would an astronaut roll over and go back to sleep or would he force himself to get up?”
For decades, Chris made choices with an eye toward space, becoming both an engineer and a test pilot. And when NASA finally came knocking, he was ready.
By the time Chris pulled out his guitar to show the world how music functions in space, he’d already spent over 160 days off-planet during three journeys, most notably as commander of the ISS. But despite spending more time in space than only a handful of humans, Chris is quick to note only a tiny percent of his actions as an astronaut took place in orbit.
So what did he do with the rest of his time?
It was divided between providing support for other missions and enormous amounts of training. Moments in space are precious and extremely expensive. NASA doesn’t want astronauts fumbling around and wasting valuable time in orbit, so every task they perform in space is first done over and over on the ground to ensure maximum efficiency.
But it’s not just the foreseeable challenges of space they drill. NASA and its partner space agencies are obsessed with a very specific type of pre-mortems: simulations. They simulate disaster after disaster that might occur in space. And many of them actually do.
Let’s go back to the 737 Max crashes for a second. Both planes had sensor failures. The pilots surely knew such errors could happen, but the Lion Air pilots didn’t know a sensor failure could trigger a secret software program to nosedive their plane. The Ethiopian Airlines pilots knew about the software, but didn’t know they’d only have ten seconds to fix it. And none of the pilots had spent any time in flight simulators practicing for this scenario.
When Atlantis went to dock with Mir, the stakes were extremely high and the margin of error exceedingly small. Fortunately, the Atlantis crew had drilled this docking repeatedly. And they hadn’t just drilled docking under ideal circumstances. Their simulations involved dozens of scenarios of what could go wrong, and the crew drilled those as well.
The disaster scenarios included sensors returning incorrect data. In fact, Chris created a cheat sheet for himself with distance vs time. Using that cheat sheet, a view from a camera at the back of the shuttle, and the second hand on his wristwatch, Chris was prepared to eyeball the approach and dock without guidance from the sensors.
The result? Chris judged the actual distance to be about 22 feet away. He and the other crew on deck coordinated the approach using their watches and eyes. In the end, they docked at the correct speed, coming in only three seconds early. But as often happens in space, they resolved one problem only to confront another. The shuttle successfully connected, but its latch wouldn’t open; it was sealed too tight. To overcome that obstacle, Chris used a non-space age solution. He pulled out a Swiss Army Knife and cut his way through.
This is neither my first nor last anecdote involving NASA. Is this because I’m some sort of space nerd? No, it’s because this is a book about failure. I don’t say that as a criticism of NASA, but as a recognition of the amount that can go wrong in space. The space agency has had its share of spectacular failures (and we’ll look at one a little later), but for every one of its actual failures, NASA’s successfully averted hundreds more.
NASA is obsessive about its simulations. They built a 6.2 million gallon swimming pool (about ten times the volume of an Olympic swimming pool) called the Neutral Buoyancy Laboratory to simulate a zero gravity environment. If a mission requires a spacewalk, they’ll drill it repeatedly in the pool.
Simulations help space agencies discover problems on the ground before encountering them under the harsh conditions of outer space. For instance, in one ISS simulation, run by the Russian team, they filled a practice capsule with so much smoke that by the time the crew got their gas masks on, they could no longer see their feet. Chris, acting according to American protocols, evacuated the crew to a part of the station sealed off from the smoked-in module where they could regroup and form a plan. But the Russians weren’t pleased with this response. They’d been trained to stay and fight a fire, to prioritize saving the lab over the crew. The simulation helped both teams identify a gap in culture and formulate a plan for handling such a scenario in the future.
Chris has died over and over again during these simulations. Most of the time, his death came about because of a poor response to a simulated disaster. But on at least one occasion, his death was the simulation. It started with a report that he was injured in space. The crew had to determine how to handle a severely injured astronaut. But before they’d gotten much farther, more bad news came in: Chris had died of his injuries.
At this point, the crew had to deal with a wide range of questions. What should they do with the corpse? There were no body bags on the ISS. Should they store it in a space suit? And what about its long-term resting place? Would they return the body to earth for a proper burial? Should they let it float off into space? Or should they hold a ceremonial cremation with the body burning up when it reentered the atmosphere?
Someone also needed to get in touch with Chris’s family. Should they deliver the news over the phone or in person? Before they could make any decisions, the scenario got worse. The press got wind of a disaster in space. How should they handle the PR? And who would take the lead, NASA or the Canadian Space Agency?
Chris’s wife had been planning to go trekking in the Himalayas during his long stint in space. After running through the simulation, she realized that should anything go wrong, she didn’t want to be on the other side of the world with limited communication and no way to quickly return home. In the end, she postponed her Himalayan adventure and went hiking in Utah instead.
Simulations can be resource intensive, but they can be fabulous tools for finding and fixing potential problems before they arise. Let’s look at a few components that can play into a successful simulation:
- Identify a clear objective. Determine exactly what you’re hoping to achieve with the simulation. Are you hoping to identify bottlenecks? Test your process under load? Make sure all systems work as expected?
- Plan your scenarios. Outline a range of scenarios that might arise, including one everything goes right, some with minor issues, and some with major issues so you can see how your team and system handle each situation.
- Create a realistic environment. The simulation should mimic the real-world environment as closely as possible, especially in the area you’re testing. For NASA, they’ll simulate spacewalks in a pool. What do you need to simulate and how can you make it as realistic as possible?
- Clarify roles and responsibilities. Each participant in the simulation should have a clearly defined role that mirrors the role they’ll play in real life. This will ensure that everyone knows what they need to do and can help identify gaps or overlaps in responsibilities.
- Define your success metrics. That which is measured tends to improve. Figure out what metrics would be most useful to measure the success of your simulation. Is it the time it takes to resolve issues? The amount of user engagement? Revenue? You want to go in knowing your key performance indicators so you can see where you need to improve.
- Test under stress. Push your systems and team to their limits to see how they handle high-pressure situations. This can help identify weaknesses that might not be visible under normal conditions.
- Gather feedback. After the simulation, poll the participants and analyze the data. What went well? What didn’t? Did any unexpected issues arise?
- Pinpoint actionable insights. The simulation is only as valuable as the takeaways you derive from it. What concrete changes should you make based on the simulation results? Do you have any systems that need to be upgraded? Does your team need additional training?
Understanding what could go wrong and planning your reactions in advance can enable you to make better, faster decisions if problems arise. While Chris Hadfield commanded the ISS, the fire alarm did actually go off. Fires in space are extremely dangerous, but no one panicked because the entire crew had drilled onboard fires. They calmly conducted checks until they found the root of the problem, which was merely a broken smoke detector. But had it been something more serious, they had the training and presence of mind to react.
The original version of David Bowie’s Space Oddity ends with Major Tom floating helplessly in space after losing communication with Earth.
Ground Control to Major Tom
Your circuit’s dead,
there’s something wrong
Meanwhile, Tom sings…
Here am I floating
round my tin can
Far above the Moon
Planet Earth is blue
And there’s nothing I can do.
Chris recorded his version immediately before returning to Earth following a five-month stint on the ISS. He changed the ending as follows:
Ground Control to Major Tom
The time is near, there’s not too long
While Chris sings:
Here am I floating in my tin can
Last glimpse of the world
Planet Earth is blue
And there’s nothing left to do
All Chris Hadfield’s years of training paid off. By preparing himself for the worst, he avoided the worst, and his simulated death in space was never repeated in reality.
Unlike checklists, simulations aren’t cheap or easy to pull off, but they can effectively prepare you against potential causes of disaster.
