System Fractures: A Summary

The causes of disasters fall into a small number of repeating patterns. These lessons apply not only to the operation of complex, high-energy systems, but even to daily lives and workplaces. As an analogy, consider how a piece of metal breaks over time. Under stress, cracks begin to grow out of tiny manufacturing flaws and damage during use; then at a critical point a crack spreads like a gunshot and the piece fails completely.

As with metal, weak points appear in all systems: these weak points are human errors and machine malfunctions. A good system is one in which people catch weak points early, before a string of them link up to a system fracture. Good systems have much redundancy, which usually takes the form of many alert people who are alert and empowered.

This is relevant because technological disasters hardly ever come like bolts from the blue ... from a single unexpected event. Nearly all have been preceded by early warning signs, called precursors. These occurred days or weeks ahead. These were indications that serious flaws existed and were starting to link up. The good news is that these precursors, if noticed and acted on, give people a chance to act before the day of disaster.

Companies and agencies that deal in high-energy, complex machines need to be reminded on a regular basis that techno-disasters have a high and long-lasting cost. Beside the obvious costs of deaths, damage, hikes in insurance costs, and months of business interruption, in some cases so much public mistrust follows that an entire segment of industry may be wiped out, once the public comes to see it as both risky and optional. At this point a disaster becomes a business catastrophe, literally, a turning point. Crashes of two Comet airliners in 1954 halted British airliner manufacturing for so long that the American industry took over the business of manufacturing jet airliners. And the scale can be enormous: A series of dam failures in China on the night of August 7, 1975, killed over 26,000 people. Incident costs ran well over four billion dollars at the Three Mile Island Unit 2 partial meltdown. When the final tally is in, response costs for the failed flood-control system in New Orleans and the Deepwater Horizon blowout will be higher still.

Broadcast Interviews: the Why and the How

Well, maybe not a lot on the "why" (writers are supposed to get out there and promote for the publishers) but here's some on the "how." If you're interested in the fun details of national media, read on!

Preparing: Generally I get a notebook together of sticky notes, organized with different subjects on specific pages, and that's my briefing book to study beforehand. I have no script; it's just responding to questions from the director, who sits alongside the camera and reminds me "don't look at the camera - look at me!". TV directors want short vivid quotes (each 5-7 seconds or less) that they can clip from a full interview tape, to go along with the narration and the video. That can take several hours of interviewing, so they're not using much! Most interviews put me in a chair, though I much prefer walking around. But that's harder to shoot and to maintain continuity so that doesn't happen often.

NPR's All Things Considered: that was a 20-minute interview about my article for Smithsonian on the shenanigans of early congressmen. They set up the interview in the basement of Central Lutheran Church in Minneapolis, I guess because Minn Public Radio's studios were booked. I was on the phone with one of the hosts, and it was recorded for later broadcast that Thanksgiving weekend.

NPR's Kojo Nnamdi Show: They got in touch through Harper's publicist, and set it up for DC when I was there on a book tour. Location was the WAMU-FM studios. A professional setup, complete with “green room” to get ready. Host was Frank Stasio, now at North Carolina public radio. He had actually read Inviting Disaster and had a lot of good questions, and listener calls. Got a picture of that. The audio is here.

TV networks: Cable TV usually hires an outside production company to do everything, after being hired by the network. The network has an exec, the showrunner, to keep tabs on the show and the budget. He or she has a final say over how it turns out, using memos called “notes” to have the prodco make changes to the rough cut.

CBS's Charlie Rose show: for an article I did about how buildings would age if abandoned. This was at the CBS studio in NYC, late at night. Had a limo driver for that one.

National Geographic Channel: Production company was out of London, Darling Smithson, doing a final show for the “Seconds from Disaster” series, on the Challenger loss. I met them in Wichita and we shot the footage at the university and at a space museum in Hutchinson KS. A larger than average crew for cable TV: six people, because they were shooting some scenes at the museum to recreate the cockpit and the commission meeting.

History Channel: because I've done over a dozen shows on a variety of series, these have been shot all over the place, from my living room, somebody's house overlooking a canyon in LA, overlooking the collapsed 35W bridge, the NYC waterfront for the Inviting Disaster series, various hotel meeting rooms and studios, and a steady rain in Toronto for the "Life After People" series. THC has a tight budget so typically in the field this is a crew of three: director-writer, a camera person, and a sound person. (David DeVries was the director on Inviting Disaster, and when we were taking a ferryboat over to Manhattan the wind was so strong it whipped the glasses right off his head, so he had to work the rest of the day without them.) Occasionally the production company adds one or two for lighting or to handle production details. Everything and everybody fits in an SUV. Learned that in NYC, the camera SUVs have steel cages to safeguard the camera equipment in the back, which works except in the case of thieves carrying thermoses of liquid nitrogen. Key role for the sound guy is to make sure there aren't distracting noises in the background. One interview was interrupted several times because of a parrot nearby, calling from a window well.

Clear Channel's Coast to Coast AM: They insist on a land line: no VOIP, no cellphone. These interviews (mostly with Ian Punnett, all live) have been as short as 10 minutes or as long as three hours.

Trapped in Chile

Those curious about the history of mine and other collapse rescues might check out my article for Invention&Technology on heavy rescue. Premise of that piece was that three needs moved the field of heavy rescue forward the most: pulling people from railroad wrecks, mine rescues, and extracting people from building collapses during the Blitz of London.

First reports suggested that everybody had been crammed in an oversized broom closet; there is such a refuge but they don't use it much and have been roaming around a mile of tunnels. I was relieved to hear about the elbow room since deep mines can flood rapidly once the pumps stop. So that's better than 33 men stuck in a hot chamber with about 12 square feet per person.

Reporters have been pondering how well the men will hold up, now that they know that rescue could take months (I'm predicting the job will go twice as fast as predicted but that's still a long time). I looked back through some historical accounts.

A collapse in Saxony, Germany, in 1963 prompted a detailed study on the effects of entrapment and delayed rescue. When 11 men waited two weeks for rescue in total darkness they hallucinated after eight days; several claimed they went for four to seven days without drinking anything because they feared the water was poisoned; and the men experienced extreme hunger for a couple of days and then that feeling went away. Doctors surmised the men could have gone for more than two weeks with only water to drink and would have suffered no permanent injury. Of course, the doctors weren't down there ...

As miners awaited rescue from a 1926 iron mine collapse in Michigan, one tried to blow himself up with dynamite but was stopped in time. Another claimed he was so desperate for a smoke he ate three corncob pipes.

Biggest lesson from historical accounts back to 1900: every mine should have at least two exits.

Sidebar on Origins of the Oppo Team

A post of political history, in honor of the primaries yesterday.

Before Karl Rove, before Lee Atwater, before Dick Morris, there was Quigley. That's Walter E. Quigley of Minneapolis: born in 1890, died in 1962. Quigley was the first to make a business out of opposition research for political campaigns. Think of oppo teams as low-profile tacticians who dig up the mud for others to sling. He opened shop during the Minnesota governor's race of 1930. His profile escalated greatly in 1946, after he helped defeat the renomination attempt of Republican Sen. Burton K. Wheeler in Montana. Quigley called his work "political dynamiting" for its sudden, shattering effect. According to a political scientist who profiled him in 1957, Frank Jonas, some incumbents paid Quigley to stay out of their states entirely.
 
Today the people carrying on Quigley's kind of demolition are called opposition research teams, or "oppo teams." An oppo team scans thousands of articles and columns bearing opinions attributable to the opponent. It videotapes his ads and speeches. It puts together searchable databases of the enemy's actions, from pardons to personal investments to roll-call votes to entries in the Congressional Record. Anything that might split off large chunks from the opponent's base is of interest. It's why today's candidates do everything they can to seal records before they run for high office. Television producers depend heavily on oppo teams since they are cheaper than their own stable of researchers, as shown in a BBC documentary about the 2000 election, Digging the Dirt.
 
Grossly unfair attacks on candidates are as old as colonial politics. The earliest came as anonymous handbills, and were followed by newspaper editorials such as those drafted by Stephen Simpson in the Jackson campaign of 1824. The plausible, late-arriving attack later known generically as the roorback appeared in 1844. (It was a long letter in the pro-abolitionist Ithaca Journal, and claimed to cite an 1836 book by Baron Von Roorback, who wrote that slaveowner and presidential candidate James K. Polk had once marked forty of his slaves with a red-hot branding iron. There was no such book and no such baron, as voters learned just in time to save the race for Polk. ) Traditional roorbacks went underground in the twentieth century, and still surface in the form of whispering campaigns. For use in public settings, something more durable was needed, which led to the brisk market for oppo workers.
 
Quigley was a one-man oppo team. He began a job by spending weeks at newspaper morgues, state archives, city libraries, and the Library of Congress. He was looking for short excerpts that could be linked to some currently inflammatory subject. For that, Quigley usually leaned on the old reliables: labor vs. business, racism, anti-Semitism, anti-Catholicism, anti-Communism, and allegiance to whatever war the U.S. was pursuing at the time.
 
Quigley used the worst of the blandishments to paste up a four-page newspaper that fairly shouted with banner headlines, heavy type, and vicious cartoons. His paper was timed to arrive in voters‚ mailboxes six to eight weeks before the primary or general election. During the 1950 U.S. Senate race in Utah, it was a newspaper-like publication called United States Senate News that targeted three-time Democratic Sen. Elbert D. Thomas, a Mormon and statesman who had been viewed previously by most Utahns as a patriot fully in tune with the majority. But the Senate News convinced enough voters that Thomas was a dupe of Communist "pinks." To devestating effect, Quigley highlighted Thomas' ill-selling The Four Fears (1944).
 
In 1957, Quigley offered this advice to young dynamiters: high-profile politicians and opinion makers are more vulnerable than newcomers because they have left a trail. One of the men he had in mind was Theodore Christianson, a Republican primary candidate in Minnesota. As part of his successful attack on Christianson in 1930, Quigley harvested quotes from hundreds of reactionary editorials that Christianson had written for his family-owned newspaper. These were enough to alienate thousands of Catholics.
 
Though called a "Republican Party cheap-shot artist" by a Utah paper in 2005, Quigley was blind to principle. Quigley took money from progressives, monopolists, red-baiters, machine Democrats, war-hawks, and isolationists from California to New York. Four years before his attack on Sen. Thomas as a Communist sympathizer, Quigley had attacked the nomination campaign of Republican Sen. Burton K. Wheeler, but from the opposite direction. According to Quigley's Montana News, Wheeler's speeches showed him to be dangerously hostile to the Red Army, which had so nobly resisted the Nazis and paved the way for Allied victory.
 
Further casting doubt on the Republican tag, Quigley was a radical long before he turned mercenary. He began his career as an agitator and recruiter for the Farmers Nonpartisan League, which was passionately socialist and so effective (at first) that the League took control of North Dakota's state government in the 1916 election. But resistance grew. The establishment saw the League as pro-German, and some League gatherings in Minnesota triggered attacks over the next two years. One of the most outspoken champions of the League was former congressman Charles A. Lindbergh, who had its backing while running for the Minnesota Republican gubernatorial nomination in 1918. Quigley was a key campaign worker for Lindbergh (father of the flier) and therefore was the target of violence by anti-League forces.
 
It was during Lindbergh's run for Minnesota governor that Quigley learned about the power of excerpt-driven attacks. Students of that election attribute Lindbergh's defeat to two periodicals circulated by anti-League forces. Both publications drew on snippets from Lindbergh's congressional speeches and his anti-war book, Why Is Your Country At War? That 220-page book was so inflammatory that Attorney General A. Mitchell Palmer had the FBI track down all known copies and lock them away as dangerous to the war effort.

That Deepwater Mentality, Pt. 2

I've been interested about why decision-makers at BP, Transocean, and the ol' MMS felt that deepwater blowouts were so unlikely, or at least acted that way when outsiders inquired about them. From my article on deepwater drilling in the Gulf for Smithsonian in 2001:

“According to Larry Flak, vice president of Boots & Coots/International Well Control, the risk of a disastrous deepwater blowout is very low, given both the precautions taken and the physics involved. A blowout preventer stopped the giant P-36 rig off Brazil from causing a crude-oil spill onto the seafloor when the platform capsized and sank in March 2001. Flak, who headed up the 1991 oil field fire-fighting effort in Kuwait after the Iraqi retreat, says there has never been a deep-water oil-well blowout. Depth is an ally: in shallow water, escaping petroleum shoots straight to the surface from the seafloor, engulfing the rig in explosive gases or, worse yet, a firestorm. But in a mile-deep blowout, water pressure would help staunch the leak from the well; the slow sideways current of the open ocean would then sweep the oil and gas far from the rig, where it would disperse long before it could reach the surface. That would allow the blowout repair crew to get to work without delay, operating from the rig itself.

“But Richard Charter, marine conservation advocate for Environmental Defense, believes that the risk of a large oil spill from deepwater wells is real. 'Conditions at this depth are poorly understood,' Charter says. Citing trouble the industry has had in pinpointing the source of leaks in shallow-water equipment off California and Alaska, he predicts that deepwater leaks will be difficult to locate and fix using remote control technology. Deepwater oil spills could drift long distances on undersea Gulf currents before surfacing, perhaps coming to rest on the sugar-white beaches of Florida's -west coast. 'Anybody saying there aren't increased risks to the environment from deepwater drilling isn't paying attention to the facts,' he adds.”

After the Deepwater Explosion I went back through the pre-blowout trade literature, and it struck me that Larry (whom I have a lot of respect for) might have been somewhat circumspect in his answer. While there was a general feeling in the industry that boreholes would bridge over in the case of a high-pressure blowout and therefore be self-sealing, he did have some concerns. In fact in January 1997 he wrote a perceptive article for Offshore Magazine on the risks of deepwater blowouts, and how to manage them.

He ended his article with this chilling message:

“Blowout control options in ultra-deepwater are very limited. Blowout prevention is of paramount importance.”

I visited his shop and interviewed him later for my Inviting Disaster book, on lessons in personnel safety from oilwell firefighting. A great interview subject!

Unfortunately for the industry and the Gulf, Larry – having died in a catastrophic boating accident in 2009 -- was not available to consult on blowout prevention during the drilling of the troublesome Macondo 252 prospect well, or on the emergency response. Our loss!

That Deepwater Mentality, Pt. 1

The factor called "time to first oil" is a key concept in drilling economics, in some way more important than how much cash is spent.

Two things related to blowout preventers have concerned people who are trying to reduce time to first oil:

1) the time spent in testing BOPs on station; and

2) what to do when a BOP fails a pressure test.

Because the BOP is on the critical path for deepwater drilling, and deepwater MODUs and drillships typically have no replacement subsea BOPs on board, a given project can lose the better part of a week tripping the BOP up and then back down from mile-deep waters. The following patent describes a response to the first concern, how to shorten the testing time with computer methods.
http://www.patentstorm.us/patents/7401654/description.html

One item I look for when reading through the disclosed correspondence between Transocean, BP, and Cameron are concerns referring to the "MUX."

The Deepwater Horizon had a "MUX" style blowout preventer, MUX standing for multiplex control, meaning that the rams and shears were electronically controlled from the surface rather than relying on hydraulic lines. A typical MUX ram is dependent on stored energy down below (in the form of pressurized fluid inside the “accumulator bottles”).

Reading the industry literature circa 1997-2001 suggests to me that the operators like Transocean went to the MUX approach because it allowed faster disconnects than hydraulically controlled BOPs (fast disconnects can be important given that dynamically positioned ships sometimes drive off or drift off accidentally), and also it was supposed to cut maintenance costs. Also, MUX control lines are cheaper than hydraulic control lines.

While industry ads of the time talked much about the greater safety of the new MUX BOPs, according to the quality assurance paper on the House Energy committee website the MUX had its own set of failure modes, such as cable-connector failure. Ironically, the MUX controlled BOP was supposed to be a lot more robust than a hydraulically controlled one because it would instantly reveal failures down below, indicate the fix needed, and specify which parts were necessary.

From what I can tell the BOP on the Deepwater Horizon was a fairly early model in the MUX evolutionary tree, so it might have suffered more age effects than later models.

BOPs are custom built for a ship and so expensive that spares aren't kept ready to hand. Operators are reluctant to replace them, and will go to great lengths to recover BOPs accidentally dropped from a ship or floating platform, or from the end of the riser, and that landed in deep mud on deep seabeds. Even if it takes a month they have undertaken such efforts, at great expense -- one vessel that lost a BOP, and later found it, was the SAIPEM 10000.

Opening the windows

An idea I mentioned in my disaster book, and one that I usually include in lectures when talking about learning from close calls, is the "bust.” That's the name code breakers used in World War I when the Germans goofed up in the process of switching from one monthly cipher book to a new one. Sometimes an operator on the German front would realize he'd just sent a message in the old code (one the British had cracked after a couple of weeks of work). Occasionally he'd make the grievous mistake of resending the message in the new code (one that the British hadn't cracked yet). The bust was a Rosetta stone for British and American code breakers. It saved them hundreds of hours of work that otherwise would be required at each replacement of a code book.

So in general, think of a bust as a unique, unexpected event that opens a view into a secretive world that could be a money laundering network, Enron traders, Wall Street derivatives, a Ponzi scheme, or an arcane industry that had virtually no outside scrutiny beforehand, like deepwater drilling.

Disclosed emails are the most reliable source of busts these days (eg, the "Enron email corpus," still studied by academics), but busts also come in the form of a briefcase of files found in a plane crash, the testimony of Alexander Butterfield during the Watergate hearings that revealed Nixon's secret taping system, a tractor that fell into a smuggling tunnel in California, and many more. I've collected accounts of hundreds of busts over the last five years.

A recent example of the bust is the discovery of two interrogation videotapes under a desk, made in 2002 at a CIA-financed prison in Morocco. Until this disclosure, the official story was that no videos survived from the rendition days.

There are more busts to come as hundreds of thousands of pages of documents from the Deepwater Horizon blowout reach daylight. Already the emails and letters and meeting memos tell us about previous close calls, neglected maintenance, and internal debates.

And there are many more leads out there, waiting for some kind of breakthrough. One of the strangest was a string of fires at an Iron Mountain records storage facility in New Jersey, in 1997, never solved, that strongly indicated some group was extremely serious about destroying some company's files. So serious about it, that there were three fires in ten days ...

Visiting the deepwater play, 2000

Even as the reading public shows signs of Deepwater news fatigue, there are authors rushing to get books out that have oil slicks or fireballs on the cover. I'm not one of them: I confined myself an article for Invention & Technology magazine on the history of ROVs. It's in the current issue.

Until full investigations with subpoena power are complete, revelations are likely to be slanted one way or the other by selective releases.

I've been interested in the subject of deepwater drilling, and its benefits and risks, since 2000. In December that year I had the chance to spend four days on a deepwater oil & gas exploration project in the Gulf of Mexico for a Smithsonian article, published the following year. Coincidentally the ship I visited was the Discoverer Enterprise, which has been the base for the response work over the summer.

The rig owner was Transocean Sedco Forex; the “company man” on board was a BP employee. Also coincidentally, the time slot allotted for my visit was when the drillship was finishing up a deepwater confirmation well – the same stage at which the Deepwater Horizon met its end. As with the well at the Macondo prospect, the well I visited had tapped a very productive deposit, at what is now called the Thunder Horse field. Finally, the ship encountered a problem with its blowout preventer when I was there – not a direct safety issue, but a costly and frustrating refusal of the BOP to unlatch from the wellhead. This problem locked the Enterprise to the site for over a day and kept it from moving to its next job. I never heard a solid explanation for it when I was there, but there were theories, like methane hydrate gumming up the latching mechanism.

I mentioned this glitch in my article, along with a few mishaps aboard the Enterprise during its shakedown phase, such as a few tons of drill collar that crashed through the top of the control cab before I came. The mention of any mishaps, and my short discussion of the risk of deepwater blowouts, apparently upset Transocean when the article came out and it struck me that the deepwater industry was not accustomed to outsiders commenting on problems of any kind.

But let me add that Transocean when I was there looked to be genuinely focused on employee safety, including use of protective equipment, thorough provisions for emergency rig evacuation, and safeguards against helicopter crashes on the helideck. I still hold the handrails when I'm using stairways, based on the mandatory and persuasive safety briefing from the Transocean medic when I arrived.

The Other Book

Helicopter enthusiasts: check out my webpage on my book for Random House, The God Machine: From Boomerangs to Black Hawks, the Story of the Helicopter. It's a social history of helicopters, inventors, and pilots. It came out in 2007 and the paperback followed in 2008. Had a great editor on that project, John Flicker of Bantam Dell.

I got interested in the subject of what helicopters could, and couldn't, do while researching a chapter to add to Inviting Disaster on the World Trade Center evacuation. Scientific American named it as a holiday book pick for 2007.

Japan's Take on Emergency Prep

Back from Japan: spent a week there, touring Tokyo, Kyoto and Hiroshima. Many things were strikingly different. One are their emergency preparations, some better than ours, some not.

Most buildings I saw that were taller than ladder-truck height had one set of external stairs, so evacuees wouldn't be smoked out by an enclosed stairway turning into a chimney. The hotel we occupied in Kyoto had what appeared to be fire refuges outside of each window, something like a very small patio.

I was surprised to see heat sensitive elevator buttons in the lobbies there: I don't see them in US high rises anymore because of concerns that fires will bring elevators to the fire floors. Maybe they have a software override?

Earthquake preps are better in Japan after the widespread death and destruction around Kobe (1995) reminded them that much more work was needed to prevent deaths from building collapse (particularly the collapse of wooden buildings with heavy tile roofs) and widespread fires.

Seismically activated gas-shutoffs are required now.

Japan has earthquake simulator rooms to teach kids how to respond when the Big One comes. Dial it up, and anybody inside can see that these events are something to take seriously. (One of these machines, which fits on a truck, was used in the Oakland area for a while doing similar demonstrations at schools, but there was too little support here to keep it in operation.)

The hotels all had flashlights by the beds and I saw instructions like these: in case of earthquake, open the room door so it won't jam shut as the building racks, and trap you inside.

On a cable car at Yomiuri Land (an amusement park in the western suburbs of Tokyo) there were instructions in case of mishap, in English, headed “In the Wish of the Emergency.” Striking phrase!

We found they had gone to great effort to offer signage and audio announcements in English as well as Japanese. We did see a trend to offer Chinese as well and I predict that will grow … a lot.
We rode the bullet trains (shinkansen) several times: found them roomy, fast at +120 mph, and they run like clockwork. It's not a cheap way to travel for residents, but the Japan Rail pass available to visiting foreigners makes it very affordable. Japan Rail can be used on JR buses, subways, local trains, and the bullet trains except for the Nozomi routes.

Toyota and sudden acceleration: needed better reportage

After too much deference to plaintiffs' lawyers, reporters lately have begun giving a more balanced treatment to this story, as “black box” information begins to reveal that some cases most likely arose out of the driver's pedal confusion.

While researching Inviting Disaster, I interviewed Bob Young of the NHTSA about sudden acceleration cases and he said that upon close investigation the majority of cases showed no mechanical malfunction and were likely due to the driver putting his or her right foot on the gas pedal while thinking it was the brake pedal. Then they froze up as a horrifying set of crashes followed; panicking so thoroughly that they didn't take even a simple action like turning the ignition off. One point that Young made struck me in particular: if while sitting on the street, somehow a car's throttle really had moved itself into the full-power position and then had stuck there (unlikely but perhaps not impossible in a drive-by-wire electronic system), the driver's foot if pushing fully down on the brake pedal at the same time (as drivers claim they did) would have overpowered the car's tendency to lurch forward. That's because the brakes in a street-legal car have more ability to decelerate a car than an engine has to accelerate it.

I do agree that drive by wire systems have many more failure modes than mechanical linkages so getting to the bottom of this is a priority.

Pointer: if SAI happens to you (and it's more likely after age 55), turn off the ignition as you hit the brakes. It's the fastest and surest way to stop the car.

35W anniversary - loose ends?

Thoughts on third anniversary this month, of collapse of the 35W bridge over the Mississippi River crossing, Minneapolis: I respect the work of the NTSB but still feel two factors were given insufficient attention in the final findings. 

The report focused on understrength gusset plates as built, and how repaving work had concentrated static loads on certain parts of the bridge. Also worthy of study:  truck traffic on the bridge that afternoon as an initiating event, not necessarily because any trucks at the time were overweight. Pavement roughness has been known for decades to be a major stressor when heavy trucks are crossing a bridge, and that day there were a couple of sizable bumps that resulted from pavement not milled off near the expansion joints. My point: even a two-inch bump on the pavement can put a lot of stress on the bridge if traffic speeds are not held to a crawl.