A Traveler's Tale, Part 3: The Root-Cause Conclusion

Part 1 and Part 2: Rick Asplundh plans an easy test of the company's time machine at the request of the R&D Division chief, intending to go back in time just a few minutes and then return. But the uncalibrated machine goes back much further, dropping him into the Black Hills outside Deadwood, Dakota Territory. And the “return to home” button doesn't work either. 

Asplundh waits by the stranded machine, then goes into town to find work. He takes a job at a gold mine tending mules underground, and quickly learns that the underground workings are deep in crisis: Earlier that month, something very odd started afflicting the mules and the mining gear too, reducing daily tonnage, and then ore quality. Seeing the miners spiral into despair, Asplundh realizes he has a skill nobody in 1877 has: training in root cause analysis! 

Co-opting the Acme mine's baseball team to serve as fellow investigators, Rick uses all the tools in the root-cause-analysis toolbox to narrow the huge initial brainstormed list of possible causes. He feels he is closing in on the chain of events until one fateful day, when the Acme mine's management decides that he has been planning to sabotage the operations for a rival owner.

Now, Part 3 of the memo from our hero to his far-away boss, “A Traveler's Tale.”

                         ~  ~  ~  ~  ~  ~ ~  ~

THIRD MEMO

From: Richard C. Asplundh, Staff Engineer, Rapid Prototyping Div.

To: Boss

Re: Mixed Results with Prototype X-1A Time Machine, Continued

Date: September 8, 1877

Via: Sulphur Springs Tonic Water

As you recall, I was in hot water earlier this week because I had my baseball team work up an Anticipatory Failure Determination. (Note to self: I should have gotten the mine bosses on board with AFD first, which is a powerful tool but one that can sound pretty strange to a stranger.)

As you know, AFD asks experts to brainstorm how they'd make sure that some specific and catastrophic problem comes about.

I'm summoned to the mine's front office and accused of planning to sabotage the mine. Why? Because I had asked the company baseball team (doubling as my informal root-cause team) to work out answers to the following question, while hewing to the principles of AFD: “If you wanted to guarantee that the miners in Acme's 'Queen of the North' payzone go after the worst ore possible -- ore so bad that it would crash the whole Acme business -- how could you be sure they do it wrong every time, day after day?”

By turning the tables on the usual prevention-oriented questions, AFD has a way of prying out new and creative thinking on cause and effect. It can be very useful when departments are on the defensive and no longer contributing.

So the mine bosses prepare to wring out a confession. I'm guessing they think I've been sent from a rival mine to ruin their business, so the enemy can buy it for a song.

Time to show my cards. “Let's think this out first," I say, displaying a steely-eyed confidence I don't yet feel. I point to the calendar on the wall. "The problem started two weeks before I even landed here,” I say, "so I didn't have anything to do with it. Yep, yep, it was an odd question and I should have cleared it with you gentlemen first, but it was just my way of confirming what's really wrong with the mine's bottom line. What's more, we found the root cause! A few simple steps, and the Acme Mine's problem is solved!”

That brightens them up enough to let me pace the floor while I talk. The superintendent had been fingering a length of rope, perhaps planning to make a hangman's knot, but he sets it down. No noose is good noose!  

Cautionary note: at this point I actually haven't nailed down all the details, but it's no time to dither. I figure that just talking through the facts one more time will make the picture clearer. So I take a deep breath and reach over for the long roll of butcher paper holding my team's last two weeks of work, including the Current Reality Tree. It's damned impressive – it'd run across the street and into the saloon if I rolled it all out.

I point to a date in the timeline. "Your problem started when the Health & Safety person, Yosemite Jack, decided to get rid of the mine-rats by starving them out,” I say. “It had some unintended consequences. On or about July 28 Jack ordered the miners to stop feeding the rats the scraps from their buckets when on lunch break in the mine. The men weren't so sure about this, but they decided to go along. The rats got hungry and hungrier. What to do? Our furry little friends started nibbling on the leather steam hoses.

"The steam leaks got worse and worse, so the steam drills didn't work like before. The daily tonnage went down just a little, day by day. Look at this line right here, the first week of August,” and I jabbed at the chart. “That's from the Queen of the North payzone. The problem starts to really hit home on August 8, when some goofball spills a big pail of lard on eight crates of dynamite sitting in the sun, before it's taken down the hoist for storage in the mine. How did that happen?”

“The lard was to grease the axles on the mine cars,” says a shift boss, “and I guess somebody from the Supply Department knocked the keg over when horsing around. It got hot, the bung comes loose, it dripped out.”

“Aha! From little acorns, great oaks grow! Nobody thinks to clean the lard off and two hundred pounds of yummy dynamite go down to the bottom of the mine for storage in the powder magazine. The rats sniff out the lard, like icing on a cake. Unfed, unloved, they start dragging sticks of dynamite off to nibble them later. They hide them … where?” I point to a diagram of the mine. My finger starts at the powder magazine and slides over to the underground mule stable, a few hundred feet down a side tunnel.

Too-Tall Johnson offers: “In the mule feed?”

“Exactly! Rats live in hay because it's warm. Rats are always hiding food and they hide some in the hay for future meals. The mules come along and munch them up along with the hay and it gives the mules a bad headache. Worse: one mule bites down a little too hard, and away it goes. Now we have sick mules with headaches, we've got superstitious miners, we've got an unexplained explosion in the stable, we've got leaky steam hoses, we've got drills that underperform, and we've got a dynamite shortage. 

"So why does this go on? Nobody puts the pieces together, and some of the pieces are trouble to even talk about. We root-cause people see this all the time: the facts are there but nobody puts them together.

"Take the powder monkeys. They don't say anything because they know they're short on the dynamite count, and they don't want to let management know that somehow they're a couple of cases short. Nobody wants to speak up - they might get accused of stealing dynamite for some gang of highwaymen, or for some cheapskate mine wanting to cut its costs. What to do? They can hold back on how much dynamite they put into the holes before blasting. That seems to make sense because the holes aren't so deep as before, what with the drills losing power from leaky steam hoses.

"So now neither the drills nor dynamite are working like they did before, and the daily tonnage really starts to fall off the cliff. Look at this line on the chart here, when you managers order them to bring the tonnage back up, or else. What's the only way to bring the tonnage up?” I point to my Anticipatory Failure Diagram.

The bookkeeper speaks up. “They could have tried to fake the scalehouse numbers, but that's double-checked at the stamp mill and outside our containment zone.” He stands up to read better, “… Or they could start blasting out crumbly ore that's weak and easy to break out, whatever the assay! That would bring the tonnage up, but at the cost of quality.”

“Exactly! Each department tried to paper over their own problem, just enough to stay out of trouble. So we get the Undesirable Effects box, listed right here at the top of the Current Reality Tree: First a big drop in tonnage, then a drop in assayed ore quality. The men in the Queen of the North payzone went after low-quality quartz that's all rotten and fractured. The mining engineer was out all this time with broken bones so he couldn't get around to see what was going on. 

"He wasn't a root cause of the problem, but when he fell down the shaft and got laid up, he opened the way for the little causes to join up. So: we talked about sick mules, exploding hay, an injured engineer, leaky steam hoses, and bad cases of Giant Powder. None of these were a root cause, but they opened the way for the mine rats to cause havoc with your business plan. I'd have to admit that it's the first Rat Cause Analysis I've ever worked on.”

So that's the story, boss! They gave me a rousing cheer. Better yet, they approved a project budget and appointed two action teams, one aboveground and one underground. Next day the Acme Mine got rid of all the greasy dynamite, they cleaned out the explosive hay in the mule stable, they smoked a peace-pipe with the mine rats, they brought in a temporary mining engineer, and they bought new steam hoses and drill bits. Acme is back up and hitting the numbers on tonnage and quality too. Now we'll see if they remember my closing suggestion: weekly followup for the next two months, then monthly through the end of Fiscal 1878. The landscape is littered with great action plans that fell apart because nobody in authority followed up.

Speaking of the next two months, I'm still stuck here in 1877, but I'm going to try a little Bayesian inference to troubleshoot my misbehaving time machine. I'll need a lot more data to plot, though, so that means more trial runs of the X-1A. 

So keep me on the payroll, and stay alert for more messages! My memos might come packaged up in anything from a Sumerian clay pot to a force-field bottle.

Train collision in Oklahoma: Jump or die

I mentioned in this post, Unstoppable and the Story of CSX 8888, that railroad companies don't permit engineers to leap from a moving train except in cases of imminent collision. This may have struck some as a quaint but unlikely scenario, a reference to engineer Casey Jones' last words to his fireman: "Jump, Sim, Jump!" 

But it still happens. A recent incident came with the June 13 collision of two Union Pacific freight trains near Goodwell, Oklahoma. (Photo: Trudy Hart, Guymon Daily Herald)

Each head-end engine (the locomotive in front) had two crewmen, for a total of four lives at risk.

How could two dispatched trains smash into each other in mid-morning on a straight, flat stretch of track? So far there's no explanation why the westbound train didn't get to its assigned siding in time, though railfans have plenty of speculations. 

We have only the bare facts from the NTSB preliminary statement: speed of each train, total weight (about 12,000 tons), the weather (clear and hot), and the trains' location on a main line in the UP's Pratt Subdivision. But an explanation will be forthcoming: for one, there's recorded data. While the collision destroyed black-box recorders in the lead engines, each train had a locomotive at the rear with its own event recorder, and the UP's traffic computers should have a record of what happened when.

And there's a survivor to interview. While three crewmembers died in the crash and flames, a fourth jumped from the slower of the two trains and tumbled to a stop with minor injuries. 

Knowing when to stay or go is a key part of what I call the Survivability Zone -- more on that later.

Root Cause Analysis: The Traveler's Tale, Part 2

Recap of Part 1: In the first installment about our analytically-inclined time traveler, Rick Asplundh plans an easy test of the company's time machine at the request of the R&D Division chief, intending to go back in time just a few minutes and then return. But the uncalibrated machine goes back much further, dumping him in the Black Hills outside Deadwood, Dakota Territory. Date: 1877. And the “return to home” button doesn't work either. Asplundh waits by the stranded machine, then goes into town to find work. He takes a job at a gold mine tending mules underground, and quickly learns that the underground workings are deep in a business crisis. Earlier that month, something very odd started afflicting the mules and the mining gear too, reducing daily tonnage, and then ore quality. 

Seeing the miners spiral into despair, Asplundh realizes he has a skill nobody in 1877 has: training in root cause analysis! He convenes a group of fellow miners at a saloon and takes them through the initial scoping steps. Rick used their bullet points to write up an incident description, but that's just the beginning. He needs the Acme mine management to empower a team and get behind a business-improvement process that won't be invented for another seventy years. Now, Part 2 of the memo that Rick is writing to his boss, “A Traveler's Tale.”

       ~ ~ ~ ~ ~ ~ ~ ~ ~

           SECOND MEMO

From: Richard C. Asplundh, Staff Engineer, Rapid Prototyping Div.

To:      Boss

Re:     Mixed Results with Prototype X-1A Time Machine, Continued

Date:   September 6, 1877

Via:    "Winged-Victory" Brand beer bottle

Picking up from the first memo: I was on the way to Too-Tall Johnson's office after getting off work one morning, pondering how I could get the mine bosses to take my incident description and root-cause analysis seriously.

I find the front office abuzz with sports talk. Forget the gold mine going bust: today's panic is the mine's baseball team, because the playoffs are coming up. Another mine has hired away our coach, and odds are long. It looks like the crises are just piling up into a big explosion, but I am not discouraged. Using principles from Eliyahu Goldratt's Theory of Constraints, I think I'll try an Evaporating Cloud diagram. It's one way to work through apparently unsolvable dilemmas to a win-win solution by breaking down preconceptions. I go to a dusty counter, unroll a sheet of butcher paper, and draw the boxes and arrows, Goldratt-style: The mine bosses want the business problem to be fixed but they are so discouraged they don't want to invest any effort or staff … unless it's for the baseball team. I probably could help the mine by nominating and leading a root-cause investigation team if I could get one with knowledge and implementation authority, but that's a hard sell in the 1870s. I've got a true dilemma on my hands that if left unsolved, will wreck the business and my confidence as well.

Goldratt said that every dilemma offers a way out – no exceptions. The word “team” sticks in my mind. I decide to focus my thinking on the mine's nine and its place in this puzzling picture, then Eureka!

I evaporate the cloud by injecting more details and getting past old assumptions, just like Goldratt described in his business novel The Goal. Fact: the Acme Nine needs a head coach. Fact: I coached Little League baseball. Fact: from going to the games, I know that our batting roster has almost all the mining knowledge I need, from hard-rock miners to drillers, bookkeepers to blacksmiths to the steam hoist operators. We even have a shift boss as a third bagger, and we'll need a supervisor to carry the weight when the going gets tough, as every root-cause process does. It's got a few holes but I can figure a way to fill out any knowledge gaps, or my initials aren't RCA.

I turn, call for attention, and shout these immortal words: “What if I coach the team, and finish out the season?”

A pause, then “Huzzah for Rick!,” they say. They know from my color commentary in the stands that I understand the game, and I've shown them a few pitches outside the Gem Theater that turned their heads. Or was it the beer?

“But first!” I pull out my first-draft Problem Statement and ask for some advice. After some editing around the potbelly stove, it reads like this when signed by the mine superintendent:

“Problem: Beginning in early August the Acme Mine began suffering from a mysterious, abrupt production crisis, first reflected in daily tonnage and then a declining ore assay. Together these have slashed revenue and threaten to close the operation. Goal: Find the root cause and meet or exceed revenue targets before the next quarterly report to Corporate.”

“One more thing,” I add as I fold up the paper, which is critical to scoping my investigation and pursuing implementation schemes. “We've got a problem in the infield, so I need to switch out our shortstop with a Welshman who works in the powder magazine. I'm thinking Scorch-Face Smith.” Yes, Scorch is a good switch-hitter when he's sober, the superintendent agrees, but wants to know why. I think fast and say daily exposure to nitroglycerine is going to give Scorch that extra burst of speed to steal some bases.

The real reason is that the first wave of data-gathering has given me an intuition (yes, hunches are a valid part of any root-cause investigation, if followed with data and logical cause-and-effect rigor) that we can't dig up our root cause without an explosives guy on board.

Why? I hear that the powder monkeys know more about some problems down below than they have let on. And that reluctance doesn't necessarily mean they are part of the root cause. Typical for root cause investigations: somebody's always dragging their feet, or actively misdirecting the effort, and guilt may have nothing to do with it. They act out of fear their department will get blamed or maybe they'll have to do more work because somebody else screwed up. I'd say that office politics is in the top three reaons why root-cause efforts fail, along with lack of persistence and poor followup at the end of of the improvement process, during implementation.

So on top of everything else in managing my new 1870s lifestyle and running the mule team downstairs, I'm supposed to coach a pretty rough bunch into a championship baseball team. I teach them a few pitching and hitting tricks that Abner Doubleday never thought up. They appreciate the tips and that's why they're willing to indulge my root-cause work on the side. They think I'm half-addled – I can't even keep my story straight on where I came from. That's okay – whatever it takes!

I don't have to remind you that mustering the right investigation team is key: they need to have subject-matter expertise, good interpersonal skills to dig out the “who, what, when, where, why, and how” information, and finally the ability to implement a solution. Because this is looking like we have a long and tangled chain of causation that produces multiple symptoms – nothing simple here -- they need a good facilitator to keep them on track. That's me.

One of my jobs is to keep the root cause team/ baseball team in the containment zone. That means sticking to the range of relevant operations: not all events that affect business operations, but only those that they have the ability to change. So when they start blaming gold prices in Frisco, that's out of bounds. I'm not looking for cheery faces all the time: I usually find constructive disagreement gives the best results.

And there's plenty of unhappiness to keep me happy. Right away I have to a put a new man in right field since I need a really smart guy from the steam-drilling department. He's got bad eyesight and this would have caused an uproar except that our pitcher is developing quite a slider and not many balls are getting out to right field.

Two weeks pass. The baseball team has won a few games nobody expected and now it's time to introduce my newly energized baseball team to the Ishikawa fishbone diagram. “Now, I know what you're going to say, so just hold your horses,” I start out. “Scorch, you were going to say that fishbones have their detractors, but!  I like fishbone diagrams because filling out them out brings forth rich speculation about the full range of possible causes and connections. So bear with me.” These are hard-rock miners who haven't been doing a lot of diagramming on the frontier or any kind of symbolic logic for that matter, so it's not easy to sell them on the idea, but I insist that this is part of my strategy to raise team morale before the playoffs.

So after the practice each afternoon, and with a few gallons of beer that I buy, we flesh out the fishbone. You'd recognize it right away: problem statement goes on the right, linked to a spine that ties together branches representing major categories of possible causes. We settle on six fishbone categories adjusted for the setting:

• Machines: Steam drills (drills, boilers that supply steam); tools for dressing bits; tools used at the working face in the mine; Hoist gear; Mine cars; Mine rails; Timbers to hold up workings
• Men: Miners (poor training, poor safety discipline, labor unrest); Hoist operators; Mule wranglers and stablehands ( I make sure to include my department, to make the point that no one is immune from the evil eye)
• Materials used up daily: Dynamite, Coal used for raising steam, Steel drill bits, water quality
• Management: Superintendent; shift foremen
• Measurement: Assay office, scalehouse that weighs tonnage, surveys of underground workings
• Mine (Environment): Ore bodies; Temperature; Humidity; Flooding; Fire; Cave-ins

Normally I'd keep my fishbone nice and dry in a conference room with a big whiteboard, using markers and sticky notes. A good fishbone is something the Japanese keep posted in meeting rooms for months, in fact. But in the Wild West, I settled with a whitewashed wall on the back of a toolshed by the baseball diamond and that's my whiteboard. We just nail the pieces of paper to the board.

It can be hard to move the notes around on such a board but we've got to have something that keeps the paper from blowing away in the Chinook wind. Four-Finger Halloran, the second baseman, says we should move the mining engineer from “Men” to “Management” since he does the geology and his plans guide the excavations. The bat-boy, who's wanted in Texas for a murder or two, speaks up: “Yep, if the engineer was on the job! He was visiting his uncle over in the Glory Hole Mine last month, stepped on a rotten trapdoor and fell down a shaft. He's been laid up all month.” I tell them not to jump to conclusions like blaming it all on the absent engineer. But something here is worth pursuing, so I enrich this stem with some notes and move the Mine Engineer stem and all its twigs to a new spot under the Management branch. I need a nail to hold it and one of the boys obliges with a throwing-knife: it lodges in the paper and misses me. I give him a hard look but we now have a set of possible intermediate causes and will add earlier causes to them.

As I've said a million times or more, the typical fishbone diagram starts as no more than a categorized, broad-span brainstorm list based on current knowledge, in which the main categories show up as four to eight big branches. Within each branch are stems. Here, “drills” and “hoists” are stems under “machinery.” Each stem that looks promising to the RCA team deserves the addition of twigs that list the events leading to the specific problem. 

It's this later work, probing into possible causes with methods like the Five Whys, that brings the project to a good conclusion. I don't believe that using “why” exactly five times for each symptom is always the best method, and sometimes a better question is “how could,” but the boys seem to prefer the Whys. There are too many permutations to investigate to the Nth degree, but there's enough wisdom on the baseball team to pick the most promising theories for a closer look. I buck them up by saying that the odds are good that somewhere in this burgeoning list is our root cause, or causes, along with the chain of events that led to the bad events described in the Problem Statement. Now we need facts to prove, or disprove, whether the circled, top-priority causes were a factor.

“Okay. Now set aside all your guesses on why the mine is is in so much trouble,” I say. “Just go out and gather the data and let the bones fall where they may.” At first the boys struggle with this, since it could reflect badly on their department, but I compare it to a bloodhound. Who can lead a bloodhound when it picks up a scent? Nobody! The bloodhound finds its own trail.

Duly instructed if not inspired, the Acme Nine baseball/root-cause team indulges me by bringing new information after each practice. Some of it is in tabular form. I gather it up and try out a Pareto chart – trying to find the classic 20% of causes that will bring 80% of the benefit. Paretos need a heap of statistics and these are sparse along the frontier. All I have for each day are things like mining tonnages, water in the sump, steam generation, and staffing levels. The mining engineer is still out from his fall down the shaft, so I don't have the benefit of his expertise. In fact, it may be better that he stays away. The mood was pretty ugly last week, because some of the fellows thought he had sabotaged the mine.

We circle a dozen possible causes to start with, though we'll probably have to dig into more of them later. To illustrate how hard they work at this, take the Steam Drill Problem, which is just one twig under the branch labeled Machinery:

Why – Level 1 “The steam drills are under-performing by 35% when making holes for placement of explosives - Why?” They guess that it could be the drills, the driller, or the steam supply. The drill squad goes off to investigate and reports back to the full team that nothing is wrong with the drills or drillers, but the steam flow is low.

So that leads to the Second Why: “Why is the supply of steam to the drills low?” The team says it could be due to at least five causes, including the boiler machinery (such as plugged tubes, from mineral scaling), or a change in fuel supply, or efficiency of the steam lines. The “steam team” goes off to check and reports back that supply pressure is nominal at the boiler side, but strangely low at the far end of the hoses. A clue!

Now the Third Why: “Why is steam pressure low at the drills' inlet? “ Before charging off, I have them brainstorm all possible causes: there could be too much hose in the run, the hose could be leaky, or maybe it's plugged with rust or debris.” Give me numbers! I cry and the steam team hustles off check. They discovers that the hoses are leaking more than usual: an average of 2.5 leaks per 20 feet of hose. We are starting to close in on a contributing cause, I'm sure.

And a fourth Why: “Why are there so many leaks in the steam hoses?” They hypothesize as follows: the leather or rubber could be getting old, the man in charge of patching the routine holes might be falling down on the job, there could be sabotage, or something could be wearing them out faster. A newly appointed Hose Team goes off to investigate.

So you get the idea. By now everybody in the mine begins to see that using Five Whys is no shortcut, and that root-cause work is more perspiration than inspiration.

Meanwhile I'm compiling a detailed chronology for the Change Analysis. That's a complete list of events at the mine, day to day, over the last six weeks. It took a lot of work, but I like them: a chronology is one way to sift through the causes and effects, in this case, what might have driven the deterioration in production statistics. My Pareto chart tells me that there's no significant difference between ore production between the day shift and the night shift, but there was a striking change in tonnage over time, in one of three payzones where miners are working. That payzone, called Queen of the North, had been accounting for most of the revenue, and now it's way down, so the Pareto charts indicate a cause or causes should be found there.

We carry on for another week and I feel like we're closing in on it. The Acme baseball team is on a winning streak. Better yet, a plausible chain of causation is emerging out of our many Cause and Effect Diagrams and a Current Reality Tree. 

I start them working on an Anticipatory Failure Analysis. Shortly afterward the crisis hits, which it always does at some point among us root-cause practitioners. But this crisis is a little more pointed: namely, the point of a Colt .45. When I walk into the office, there's a committee waiting for me. Or shall I say a posse? I'll finish my report when and if they let me out of the hoosegow.

         ~ ~ ~ ~ ~ ~ ~ ~ ~

Conclusion in Part 3!

Howl of the Machine: Audio from Rena on the rocks

Nine months ago I was intrigued by an article from the New Zealand Herald in which witnesses aboard the wrecked Rena described unearthly howls, shrieks, and echoing rumbles as waves slowly tore the vessel apart while it lay stranded on the rocks off the port of Tauranga. 

Here's a picture of the cracked hull from NZDF:

Maritime New Zealand's Bruce Anderson said this about the sound: "I wouldn't say it's eerie, but it's quite spooky. It would be really interesting for people to hear the grinding sound being made as the two parts of the ship work together." 

The ship finally broke in half in early January. This photo from AP:

No audio was available to the public in October, but now it is.  

Here's an audio clip courtesy of Maritime New Zealand, recorded on October 24 by officials inside the ship. I think you'll find it memorable, and perhaps well suited for techno-haunted houses. Particularly striking is a rising roar not quite halfway through the clip.

Root Cause Analysis: The Traveler's Tale, Part 1

Following is an overview of root-cause analysis, in the manner of a fable ... with time machine! “Root cause analysis” is a common term in industry and in news articles like the recent one about the origins of cracks in a shield building at the Davis-Besse nuclear power reactor, but it's impossible to summarize in a few sentences so most writers just assume that readers know the method. But root cause analysis is a whole array of methods, developed over decades by many people. 

And now for "The Traveler's Tale," Part I.

MEMO

From: Richard C. Asplundh, Staff Engineer, 

 Rapid Prototyping Div.

To: Boss

Re: Mixed Results with Prototype X-1A Time Machine

Date: September 4, 1877

Via: Catamount Brand Whiskey bottle

Well, boss, the new time machine works! In one direction, anyway! I'm leaving my progress report in a bottle and burying it where I hope someone will find it in your time zone, and send it to the home office. Meanwhile, I'm staying busy back here and trusting that you still have the meter running, paycheck-wise.

Remember how you asked me to do just one little test run before lunch with the X-1A? As in, “Ricky, old boy, how about you go backward just a few minutes, and try out the Back-to-Base Homing Mode?”

I'm here to say the homing mode doesn't work yet. Also, you need to tell the lab rats that they miscalibrated the ChronoCounter, because that "couple of minutes" they dialed in was considerably more than a century.

Fortunately, because of my deep training in all forms of Root Cause Analysis, I've been able to make myself useful back here, pending some help from your direction.

Before I get down to details, here's the view from 40-K feet: I get a job in a hard-rock gold mine. I immediately learn that the boys at the Acme Mine are smack in the middle of an all-out business crisis. But it can't withstand my root-cause skills for long, and that's me without my laptop, or my notebook, or the stack of proprietary software. 

But as I always like to say, root cause analysis is more headware than hardware. Me and my rough-hewn buddies get things sorted out, against all odds.

The big adventure starts like this: There's a haze, I feel dizzy, then find myself in a mountain forest. The machine's control panel says it's just a few minutes before I left, but I can see that this doesn't look like the inside of our company's Southwest South Dakota Warehouse at all. I wait a day in case you might send a rescue squad from the future, but no dice, so I decide to trudge off and meet the natives, whoever they may be.

I have to say it's pretty exciting to go off exploring when you don't know if you're fifty years off the beam or fifty thousand. But I find a grubby town and get my bearings: I've touched down outside of Deadwood, Dakota Territory. It's August 16, 1877, about ten years into the Black Hills gold rush.

Nothing too glamorous about this side of the Old West: unpainted slab-sided buildings up high, and mud down low. I head for a “help wanted” sign, walk inside, and a thin geezer with a visor says they need a man to tend the couple of dozen Missouri mules that live in a stable at the bottom of the Acme Mine. 

Graybeard identifies himself as Too-Tall Johanson. He says that mules pull the ore cars from the heading down a little iron track, back to the main shaft, where a steam engine drags the rock to the surface. The mules live down there, never seeing daylight … like our IT people.

I tell the guy behind the counter that I don't know one end of a mule from the other. But Too-Tall hires me for two dollars a day, hands me a shovel, and says I'll pick it up. “The way things are going at this mine, it won't be for long anyway,” Too-Tall says. This is about when my fact-finding antennae go into hyper-activity.

“The owners in Frisco are about to close us down and everything else is going wrong too!” says he. “Cain't hardly understand it!” I clap Too-Tall on the shoulder and tell him that help has arrived from an unexpected direction. He shakes his head and I go off to grab some worn-out old miner togs from a heap in the back, which feel like they were hacked out of old pine shingles. I buy a carbide lamp on credit at the company store, and down into the dark I go. 

After a week I switch to the midnight shift. That way, I can give a few pointers to the mine's baseball team at batting practice after dinner. It's in the cellar, with 43 losses and 26 wins.

The mine is in even worse shape. According to the boys, things seemed to go south all of a sudden. Starting about two weeks before I drop in, gold-ore production took a nosedive first in quantity and then quality. And all of a sudden there were new, weird problems nobody had seen before. Miners are a superstitious bunch and morale took a tumble. 

I find out why nobody wanted the mule-tending job: there was an unexplained explosion in the mule stable a week back and it's made them all superstitious that another mule is about to blow up. Meanwhile, all the mules have belly aches and make a lot of noise. We are buying gallons of Brother Jubal's patent medicine and mixing it in the water trough, but it doesn't help.

I find out plenty of other things the first week. I buy some paper and a box of pencils and make a stack of notes back at the bunkhouse. Soon it's time to start my root-cause analysis, frontier-style. It's the world's first. (As I once explained at a staff picnic last year, while the core concepts behind root causes are recognizable in Aristotle's notions of moral responsibility and determinism, it didn't get going until Operations Research during World War II and the postwar study of loss control. So right now I'm seventy years ahead of the competition.) 

I initiate my work one Saturday night when I'm whooping it up with the graveyard shift in the Dirty Dog Saloon. I shove aside the shot glasses and peanut shells, pull out two sheets of foolscap with my incident description and pass it around to the boys for a review.

There isn't room for all my work product in one whiskey bottle, but it went like this:  

“Series of mishaps and problems at the Acme Mine beginning around August 2, continuing to date. Most time-critical is rapid deterioration in gold-ore production. Problem first noticed with downward trend in ore deliveries in tons/day, falling to 33% below targets. Persisted for two weeks. Tonnage recovered to acceptable range by August 12 but starting August 7, assayed ore quality at the stamp mill fell from 10 troy oz/ton to 1 oz. Monthly gross revenue from stamp mill dropped 32% year over year. Owners plan to close mine at end of fiscal.” 

  I was careful to word this like the classic I.D.: focus on describing the most serious symptom and don't point fingers or guess about solutions. Too early for that!

Well, incident descriptions are something new to the crowd at the Dirty Dog Saloon, so I buy another round of rotgut liquor and warm them up to the idea. The audience even adds a couple of bullet points, along with a bullet hole following gunplay between a tinhorn gambler and a placer miner two tables over.

As you know, my next task is the Problem Statement, describing the deviation from the desired state, and of course it's scoped to stay within our field of control. I draft a short paragraph, in declarative sentences, stating the goal to achieve. 

Now the hard part is about to begin: getting buy-in from the powers that be. Something tells me that they don't place a lot of faith (yet) in statistics and decision trees.


Which follows in the second whiskey bottle, so stay tuned for the Part 2 of "Root Cause Analysis: The Time Traveler's Tale."

Prometheus, and thoughts on interplanetary contamination

Noting this CBSNews item on the probability that flakes from a Teflon seal may contaminate core samples to be gathered with the Martian probe Curiosity rover's sample drill. Curiosity is on its way and due to make a nerve-wracking touchdown near the Gale Crater late on August 5, JPL-time.

Since Teflon molecules are largely carbon, the effect will be to exaggerate results from the tiny onboard analytical lab about how much native carbon is in Martian dirt.

Unintentional interplanetary contamination is an interesting subject because it may be difficult to manage. This post is about half of the problem, back-contamination: offworld material coming back home in a spacecraft, and posing a risk due to chemical or biological activity.

After leaving the theater, ears ringing with the volume, watchers of Prometheus might ponder how leaders of a real expedition might go explore the surface of a planet without bringing back creepy crawlies, particularly the micro-sized ones. 

The movie does not set a shining example here. Upon the explorers' return from the first trip out to alien-land, the hangar deck and apparently the crew's quarters are wide open to anything clinging to the suits' outsides. Or to the insides of the suits, given that the explorers doffed their helmets when checking out the alien dome.

In that respect the back-contamination precautions shown in Prometheus are no more advanced than the methods employed during Apollo's manned missions to the lunar surface, which all failed in keeping loose moondust from the astronauts' bodies, or from the capsule when it returned to Earth.

How did dust spread so widely? At the start of each mission, each pair of Apollo astronauts put on their suits, emptied the lunar module of air, and climbed outside to take samples, explore, and (in the later missions), drive their moon buggy around:

At the end of each EVA they climbed back inside the Lunar Module, closed the hatch, and refilled the cabin with oxygen from tanks. They ate, slept, took notes, then went back outside for another activity. What's the problem? Every trip brought more dust into the cabin. 

It didn't turn into a crisis, but it could have. Lunar dust turned out to be a safety risk and possibly a health risk as well. Lunar dust is the size of finely-ground flour grains, but sharp-edged, reactive, and greatly more abrasive. Here's a micrograph, from NASA:

It's also highly adhesive in a high-tech way. Note the hook-like protrusions, which grab onto space-suit fabric then dig in; and it carries an electrostatic charge that binds it firmly to smooth surfaces like helmets, camera lenses, and solar panels. Wiping with a cloth only scratches the surface (literally).

Once astronauts started using the moon buggy to get around, the dust problem was totally out of control as soon as a fender came loose. At that point, a rooster-tail of dust sprayed all riders. A NASA controller commented that the suits turned so dark from the buggy rides that it looked like the two men had been playing in a coal bin. Back in the LM cabin, dust worked itself into everything including the astronauts' noses and eyes.

The later expeditions found that after three extended EVA trips on the surface, the suits were played out: moving joints were grinding to a halt, and the dust was wearing through the fabric.

Some dust migrated from the LEM through the service module to the capsule, so a portion came back to Earth in an uncontrolled manner. While lunar dust didn't harbor a killer-interstellar virus, the latest plans call for trips to planets that could harbor life. So serious preventive work is needed before manned trips to Mars, and will be greatly more urgent when we dispatch expeditions to Earth-like planets with signs of life.

One solution could be the suitport:

In the suitport approach, astronauts never bring their suits back into a crew cabin or any occupied space; they climb into them from the back, working out of a shirt-sleeves environment on wheels:

I saw no suitports in Prometheus, but the gadget could be a lifesaver in the real world ... or worlds.

Superjet Crash: Flight data recorder, also found

Following up on my earlier post about the cockpit voice recorder (CVR) found at the Superjet 100 crash site on May 15. 

As predicted, the solid-state recording chip in the CVR was in good shape, despite the unit's battered appearance in news photos. Indonesia's investigative agency is running the last 20 minutes past three interpreters to put together a common transcript. (Why 20 minutes? The passenger jet had been aloft for just 12 minutes, following takeoff from Jakarta.) 

That text may be all we ever learn about the cockpit chatter; as in most crashes, there are no plans to release the audio itself.

Local residents found the FDR last week. It was in good condition under a heap of dirt. The maneuvers of the aircraft near Mt. Salak are already documented from radar tracks, but if the plane had some kind of mechanical problem, data on the FDR  should reveal that.

Meanwhile, given the news vacuum, one Russian tabloid blames American sabotage, but most articles cite alleged recklessness by the pilot to impress customers about the RJ's agility. 

Pushing the maneuvering envelope to show off a new plane wouldn't come as a big surprise, but I haven't seen a convincing explanation why the pilot would ask ATC to descend from a safe altitude so he could fly through the mist, among the volcanoes. One maneuver near the volcano was a 360-degree turn.

It was truly an odd request assuming that he knew his whereabouts, and wasn't distracted.