Comments about technological history, system fractures, and human resilience from James R. Chiles, the author of Inviting Disaster: Lessons from the Edge of Technology (HarperBusiness 2001; paperback 2002) and The God Machine: From Boomerangs to Black Hawks, the Story of the Helicopter (Random House, 2007, paperback 2008)

Wednesday, July 18, 2012

A Traveler's Tale, Part 3: The Root-Cause Conclusion


Part 1 and Part 2: Rick Asplundh plans an easy test of the company's time machine at the request of the R&D Division chief, intending to go back in time just a few minutes and then return. But the uncalibrated machine goes back much further, dropping him into the Black Hills outside Deadwood, Dakota Territory. And the “return to home” button doesn't work either. 

Asplundh waits by the stranded machine, then goes into town to find work. He takes a job at a gold mine tending mules underground, and quickly learns that the underground workings are deep in crisis: Earlier that month, something very odd started afflicting the mules and the mining gear too, reducing daily tonnage, and then ore quality. Seeing the miners spiral into despair, Asplundh realizes he has a skill nobody in 1877 has: training in root cause analysis! 

Co-opting the Acme mine's baseball team to serve as fellow investigators, Rick uses all the tools in the root-cause-analysis toolbox to narrow the huge initial brainstormed list of possible causes. He feels he is closing in on the chain of events until one fateful day, when the Acme mine's management decides that he has been planning to sabotage the operations for a rival owner.

Now, Part 3 of the memo from our hero to his far-away boss, “A Traveler's Tale.”

                         ~  ~  ~  ~  ~  ~ ~  ~

THIRD MEMO
From: Richard C. Asplundh, Staff Engineer, Rapid Prototyping Div.
To: Boss
Re: Mixed Results with Prototype X-1A Time Machine, Continued
Date: September 8, 1877
Via: Sulphur Springs Tonic Water


As you recall, I was in hot water earlier this week because I had my baseball team work up an Anticipatory Failure Determination. (Note to self: I should have gotten the mine bosses on board with AFD first, which is a powerful tool but one that can sound pretty strange to a stranger.)

As you know, AFD asks experts to brainstorm how they'd make sure that some specific and catastrophic problem comes about.

I'm summoned to the mine's front office and accused of planning to sabotage the mine. Why? Because I had asked the company baseball team (doubling as my informal root-cause team) to work out answers to the following question, while hewing to the principles of AFD: “If you wanted to guarantee that the miners in Acme's 'Queen of the North' payzone go after the worst ore possible -- ore so bad that it would crash the whole Acme business -- how could you be sure they do it wrong every time, day after day?”

By turning the tables on the usual prevention-oriented questions, AFD has a way of prying out new and creative thinking on cause and effect. It can be very useful when departments are on the defensive and no longer contributing.

So the mine bosses prepare to wring out a confession. I'm guessing they think I've been sent from a rival mine to ruin their business, so the enemy can buy it for a song.

Time to show my cards. “Let's think this out first," I say, displaying a steely-eyed confidence I don't yet feel. I point to the calendar on the wall. "The problem started two weeks before I even landed here,” I say, "so I didn't have anything to do with it. Yep, yep, it was an odd question and I should have cleared it with you gentlemen first, but it was just my way of confirming what's really wrong with the mine's bottom line. What's more, we found the root cause! A few simple steps, and the Acme Mine's problem is solved!”

That brightens them up enough to let me pace the floor while I talk. The superintendent had been fingering a length of rope, perhaps planning to make a hangman's knot, but he sets it down. No noose is good noose!  

Cautionary note: at this point I actually haven't nailed down all the details, but it's no time to dither. I figure that just talking through the facts one more time will make the picture clearer. So I take a deep breath and reach over for the long roll of butcher paper holding my team's last two weeks of work, including the Current Reality Tree. It's damned impressive – it'd run across the street and into the saloon if I rolled it all out.

I point to a date in the timeline. "Your problem started when the Health & Safety person, Yosemite Jack, decided to get rid of the mine-rats by starving them out,” I say. “It had some unintended consequences. On or about July 28 Jack ordered the miners to stop feeding the rats the scraps from their buckets when on lunch break in the mine. The men weren't so sure about this, but they decided to go along. The rats got hungry and hungrier. What to do? Our furry little friends started nibbling on the leather steam hoses.

"The steam leaks got worse and worse, so the steam drills didn't work like before. The daily tonnage went down just a little, day by day. Look at this line right here, the first week of August,” and I jabbed at the chart. “That's from the Queen of the North payzone. The problem starts to really hit home on August 8, when some goofball spills a big pail of lard on eight crates of dynamite sitting in the sun, before it's taken down the hoist for storage in the mine. How did that happen?”

The lard was to grease the axles on the mine cars,” says a shift boss, “and I guess somebody from the Supply Department knocked the keg over when horsing around. It got hot, the bung comes loose, it dripped out.”

Aha! From little acorns, great oaks grow! Nobody thinks to clean the lard off and two hundred pounds of yummy dynamite go down to the bottom of the mine for storage in the powder magazine. The rats sniff out the lard, like icing on a cake. Unfed, unloved, they start dragging sticks of dynamite off to nibble them later. They hide them … where?” I point to a diagram of the mine. My finger starts at the powder magazine and slides over to the underground mule stable, a few hundred feet down a side tunnel.

Too-Tall Johnson offers: “In the mule feed?”

Exactly! Rats live in hay because it's warm. Rats are always hiding food and they hide some in the hay for future meals. The mules come along and munch them up along with the hay and it gives the mules a bad headache. Worse: one mule bites down a little too hard, and away it goes. Now we have sick mules with headaches, we've got superstitious miners, we've got an unexplained explosion in the stable, we've got leaky steam hoses, we've got drills that underperform, and we've got a dynamite shortage. 

"So why does this go on? Nobody puts the pieces together, and some of the pieces are trouble to even talk about. We root-cause people see this all the time: the facts are there but nobody puts them together.

"Take the powder monkeys. They don't say anything because they know they're short on the dynamite count, and they don't want to let management know that somehow they're a couple of cases short. Nobody wants to speak up - they might get accused of stealing dynamite for some gang of highwaymen, or for some cheapskate mine wanting to cut its costs. What to do? They can hold back on how much dynamite they put into the holes before blasting. That seems to make sense because the holes aren't so deep as before, what with the drills losing power from leaky steam hoses.

"So now neither the drills nor dynamite are working like they did before, and the daily tonnage really starts to fall off the cliff. Look at this line on the chart here, when you managers order them to bring the tonnage back up, or else. What's the only way to bring the tonnage up?” I point to my Anticipatory Failure Diagram.

The bookkeeper speaks up. “They could have tried to fake the scalehouse numbers, but that's double-checked at the stamp mill and outside our containment zone.” He stands up to read better, “… Or they could start blasting out crumbly ore that's weak and easy to break out, whatever the assay! That would bring the tonnage up, but at the cost of quality.”

Exactly! Each department tried to paper over their own problem, just enough to stay out of trouble. So we get the Undesirable Effects box, listed right here at the top of the Current Reality Tree: First a big drop in tonnage, then a drop in assayed ore quality. The men in the Queen of the North payzone went after low-quality quartz that's all rotten and fractured. The mining engineer was out all this time with broken bones so he couldn't get around to see what was going on. 

"He wasn't a root cause of the problem, but when he fell down the shaft and got laid up, he opened the way for the little causes to join up. So: we talked about sick mules, exploding hay, an injured engineer, leaky steam hoses, and bad cases of Giant Powder. None of these were a root cause, but they opened the way for the mine rats to cause havoc with your business plan. I'd have to admit that it's the first Rat Cause Analysis I've ever worked on.”

So that's the story, boss! They gave me a rousing cheer. Better yet, they approved a project budget and appointed two action teams, one aboveground and one underground. Next day the Acme Mine got rid of all the greasy dynamite, they cleaned out the explosive hay in the mule stable, they smoked a peace-pipe with the mine rats, they brought in a temporary mining engineer, and they bought new steam hoses and drill bits. Acme is back up and hitting the numbers on tonnage and quality too. Now we'll see if they remember my closing suggestion: weekly followup for the next two months, then monthly through the end of Fiscal 1878. The landscape is littered with great action plans that fell apart because nobody in authority followed up.

Speaking of the next two months, I'm still stuck here in 1877, but I'm going to try a little Bayesian inference to troubleshoot my misbehaving time machine. I'll need a lot more data to plot, though, so that means more trial runs of the X-1A. 

So keep me on the payroll, and stay alert for more messages! My memos might come packaged up in anything from a Sumerian clay pot to a force-field bottle.

No comments:

Post a Comment